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ABSTRACT 


The  purpose  of  this  research  is  to  evaluate  and  refine  a  safety  information 
management  system  that  will  facilitate  data  collection,  organization,  query,  analysis  and 
reporting  of  maintenance  errors  that  contribute  to  Naval  Aviation  mishaps,  equipment 
damage  and  personnel  injury  using  OPNAV  3750. 6R,  Human  Factors  Analysis  and 
Classification  System  Maintenance  Extension  taxonomy.  The  target  audience  for  this 
information  management  system  tool  included  safety  personnel,  mishap  investigators. 
Aircraft  Mishap  Board  (AMB)  members,  and  analysts.  A  review  of  three  areas  was 
needed  to  refine  the  prototype  tool:  (1)  the  collection,  use  and  management  of  accident 
information,  (2)  human  error  theories  as  related  to  aviation  mishaps  and  (3)  the  design  of 
an  effective  mishap  database  tool.  A  usability  study  was  conducted  using  potential  end- 
users.  Fifteen  Naval  Aviation  Safety  Officers  and  Naval  Aviators  were  given  written 
procedures  to  navigate  through  the  prototype  and  an  exit  survey.  The  survey  responses, 
including  objective  and  subjective  responses  about  the  prototype  were  gathered.  The 
results  indicate,  that  with  proper  training,  the  prototype  could  provide  insight  into 
maintenance  errors,  which  could  be  used  to  target  hazards  and  develop  intervention 
strategies  to  prevent  future  mishaps. 
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I.  INTRODUCTION 


A.  OVERVIEW 

From  Fiscal  Year  (FY)  1951  to  1999  Naval  Aviation  has  had  great  suecess  in 
substantially  reducing  its  Class  A  Flight  Mishap  (FM)  rate  (see  Figure  1).  Even  with  this 
accomplishment,  the  proportion  of  mishaps  attributed  to  human  error  has  remained 
relatively  constant  at  80  percent  (Nutwell  &  Sherman,  1997).  In  1996,  a  Navy  F-14 
Tomcat  crashed  shortly  after  taking  off  from  Nashville,  Tennessee  killing  both  aircrew 
and  three  civilians  on  the  ground.  Because  the  cause  of  this  mishap  was  exclusively 
human  error,  Department  of  the  Navy  (DON)  leaders  chartered  a  Human  Faetors  Quality 
Management  Board  (HFQMB)  to  reduce  mishaps  by  identifying  systemie  improvements 
to  enhance  performance  and  systems  that  guard  against  error.  The  HFQMB ’s  goal  was  to 
reduee  human  error  in  the  Naval  Aviation  Class  A  FMs  rate  by  50  percent  at  the  start  of 
FY  2000  and  to  further  reduce  it  by  another  50  pereent  by  FY  2006  (HFQMB,  1997). 
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Figure  1.  Naval  Aviation  Class  A  FM  Rates  for  FY  1951-1999 
(From  NSC,  2000) 

The  HFQMB  adopted  a  three-prong  approaeh  to  taekle  human  error.  The  first 
thrust  was  to  conduct  an  extensive  mishap  data  analyses  foeused  on  human  faetors.  The 
Naval  Safety  Center  (NSC)  developed  the  Human  Factors  Analysis  and  Classification 
System  (HFACS)  to  eapture  airerew  errors  in  Naval  Aviation  mishaps.  This  taxonomy 
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identifies  areas  for  potential  intervention  by  fully  describing  factors  that  are  precursors  to 
accidents.  HFACS  identifies  both  active  failures  and  latent  conditions  within  four 
categories:  (1)  unsafe  acts,  (2)  pre-conditions  for  unsafe  acts,  (3)  unsafe  supervision, 
and  (4)  organizational  influences  (DON,  2001).  NSC  adopted  HFACS  for  analyzing 
aircrew  error  in  Naval  Aviation  mishaps  and  targeting  appropriate  areas  (DON,  2001). 

Naval  Aviation  achieved  its  lowest  Class  A  FM  rate  in  FY  1999  partly  due  to  the 
efforts  of  the  HFQMB.  Even  with  this  reduction,  the  HFQMB  failed  to  achieve  the 
desired  50  percent  reduction  in  human  error  (NSC,  2001).  A  study  noted  that  HFACS 
could  be  extended  to  cover  maintenance  errors;  hence,  HFACS  was  adapted  to  classify 
maintenance  errors  (Schmidt,  Schmorrow,  &  Hardee,  1997). 

The  maintenance  extension  (ME)  of  HFACS  contains  four  human  error 
categories:  (1)  Management  Conditions,  (2)  Working  Conditions,  (3)  Maintainer 
Conditions,  and  (4)  Maintainer  Acts.  A  review  of  470  Naval  Aviation  Mishaps  by 
Schmorrow  (1998)  determined  the  HFACS— ME  taxonomy  was  an  effective 
classification  for  determining  trends  in  maintenance  mishaps.  Building  on  Schmorrow’ s 
research.  Fry  (2000)  developed  Maintenance  Error  Information  Management  System 
(MEIMS)  for  the  analysis  of  maintenance  related  mishaps.  MEIMS  lead  to  a  refinement 
of  HFACS— ME,  and  made  the  data  more  comprehensive  and  accessible. 

Fry’s  rudimentary  MEIMS  was  further  refined  by  Wood  (2000)  and  developed 
into  a  working  prototype  for  fleet  test  and  evaluation.  A  usability  study  of  the  MEIMS 
prototype  determined  it  could  not  only  be  effective  system  in  determining  trends,  but  also 
providing  information  for  mishap  prevention  efforts.  Wood’s  study  identified  a  need  to 
incorporate  HFACS— ME  definitions,  improve  the  user  interface,  simplify  data  entry 
procedures,  and  provide  example  mishap  scenarios.  The  MEIMS  tool  was  further  refined 
by  McCracken  (2000),  and  training  incorporated  a  user  tutorial.  His  study  revealed  that 
participants  in  the  tutorial  group  performed  better  using  MEIMS  than  the  non-tutorial 
group.  Recommendations  from  McCracken’s  study  include  better  main  menu  navigation, 
improved  data  error  checking,  and  making  and  the  tutorial  available  over  the  Internet. 

This  thesis  is  part  of  ongoing  effort  to  study  the  feasibility  and  utility  of  MEIMS 
as  a  tool  for  investigating,  collecting,  and  analyzing  maintenance  mishaps  and 
maintenance  error.  By  further  refining  MEIMS,  it  is  contended  that  the  end  user  will  be 
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able  to  easily  access  valuable  safety  information,  which  can  be  used  in  training,  hazard 
identification  and  trend  analysis  to  prevent  possible  future  mishaps. 


B.  PURPOSE 

The  intent  of  this  study  is  to  refine,  expand,  and  revaluate  the  MEIMS  tool  to 
facilitate  the  collection,  organization,  query,  analysis,  and  reporting  of  maintainer  errors 
that  contribute  to  Naval  Aviation  maintenance  mishaps.  The  goal  is  to  provide  an 
effective  tool  to  promote  the  use  of  HFACS— ME  as  part  of  the  revised  Naval  Aviation 
Safety  Program  Instruction  (OPNAVfNST  3750. 6R). 

C.  PROBLEM  STATEMENT 

In  order  to  continue  to  reduce  the  annual  mishap  rate,  Naval  Aviation  leadership 
is  expanding  the  focus  of  their  safety  initiatives  to  encompass  maintenance  errors.  The 
systematic  analysis  of  maintenance  mishaps  offers  an  increased  opportunity  to  reduce  the 
mishap  rate,  save  lives  and  assets,  as  well  as  increase  fleet  readiness.  The  HFACS— ME 
as  it  is  incorporated  into  MEIMS  provides  a  well-designed  information  management 
system  to  effectively  identify  maintenance  error  patterns  and  trends.  The  current 
prototype  MEIMS  is  a  valuable  tool;  however,  it  needs  to  be  refined  and  enhanced  to 
capitalize  on  current  technologies  and  to  include  mishap  investigation.  This  thesis 
examines  the  following  questions: 

1 .  How  can  MEIMS  be  used  to  facilitate  preliminary  mishap  investigation? 

2.  How  will  investigators  use  this  tool? 

3.  What  processes  are  needed  to  capture  maintenance  error  under 
OPNAVINST  3750.6R? 

4.  What  enhancements  are  needed  to  make  MEIMS  more  user  interact! ve/friendly? 

5.  Could  this  tool  be  Web-based,  making  it  more  easily/widely  accessible? 

An  effective  information  management  system  will  give  the  fleet  users  the  ability 
to  quickly  access  standardized  error  data  relating  to  aviation  maintenance  mishaps. 
Providing  an  easy-to-access  error  database  will  ensure  standardization,  as  well  as  increase 
the  validity  and  reliability  of  the  data.  Ready  access  to  error  data  will  allow  maintainers 
and  safety  personnel  to  quickly  identify  potential  hazards,  analyze  trends,  and  ultimately 
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train  personnel  to  avoid  future  occurrences,  thus  reducing  aircraft  mishaps.  This 
reduction  in  mishaps  can  save  lives,  aircraft,  and  equipment. 


D.  SCOPE  AND  LIMITATIONS 

Fleet  personnel,  primarily  consisting  Aviation  Safety  Officers,  were  tasked  to 
evaluate  the  prototype  MEIMS  tool.  The  prototype  is  intended  to  be  refined  for  use  by 
Naval  Aviation  squadrons,  but  may  have  some  crossover  use  by  other  Military  Services, 
government  organizations,  and  the  private  sector.  This  study  only  focuses  on 
maintenance  mishaps  caused  by  human  error.  Material  failure,  maintenance  hazards,  and 
persormel  injuries  not  reaching  the  threshold  of  a  mishap  were  not  used  within  this  study. 

E.  DEFINITIONS 

This  study  uses  the  following  abbreviations,  terms,  and  associated  definitions: 

Aircraft  Mishap  Board  (AMB).  Group  of  officers  appointed  to  investigate  and 
report  on  an  aviation  mishap  (DON,  2001). 

Aviation  Safety  Officer  (ASO).  Principal  advisor  to  Naval  Aviation  squadron 
commanding  officers  on  all  aviation  safety  matters  (DON,  2001). 

F-14  Tomcat.  US  Navy  aircraft.  Two  aircrew,  two  engines,  swing- wing, 
supersonic  fighter  with  air-to-air,  air-to-ground,  and  reconnaissance  capability  (Rowe  & 
Morrison,  1973). 

Human  Factors  Analysis  and  Classification  System  (HFACS).  System  designed 
to  help  analyze  Naval  Aviation  mishaps  focusing  on  aircrew  error  (Shappell  & 
Wiegmann,  1997). 

Human  Factors  Analysis  and  Classification  System— Maintenance  Extension 
(HFACS— ME).  HFACS  adaptation  to  classify  causal  factors  that  contribute  to 
maintenance  mishaps  (Schmidt,  1996). 

Human  Factors  Quality  Management  Board  (HFQMB).  Established  by  Naval 
Aviation  senior  leadership  to  reduce  human  error  involvement  in  Naval  Aviation  Class  A 
flight  mishaps  (HFQMB,  1997). 

Maintenance  Error  Information  Management  System  (MEIMS).  Prototype  error 
management  tool  examined  in  this  thesis  (Wood,  2000). 
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Mishap.  A  Naval  mishap  is  an  unplanned  event  or  series  of  events  directly 
involving  Naval  aircraft,  which  result  in  $20,000  or  greater  cumulative  damage  to  naval 
aircraft,  other  aircraft,  property,  or  personnel  injury  (DON,  2001). 

Mishap  Categories.  Naval  aircraft  mishap  categories  are  defined  below  (DON, 

2001): 

Flight  Mishap  (FM).  Those  mishaps  resulting  in  $20,000  or  greater  DOD 
aircraft  damage  or  loss  of  a  DOD  aircraft,  and  intent  for  flight  for  DOD  aircraft 
existed  at  the  time  of  the  mishap.  Other  property  damage,  injury,  or  death  may  or 
may  not  have  occurred. 

Flight  Related  Mishap  (FRM).  Those  mishaps  resulting  in  less  than 
$20,000  DOD  aircraft  damage,  and  intent  for  flight  (for  DOD  aircraft)  existed  at 
the  time  of  the  mishap,  and  $20,000  or  more  total  damage  or  a  defined  injury  or 
death  occurred. 

Aircraft  Ground  Mishap  (AGM).  Those  mishaps  in  which  no  intent  for 
flight  existed  at  the  time  of  the  mishap  and  DOD  aircraft  loss,  or  $20,000  or  more 
aircraft  damage,  and/or  property  damage,  or  a  defined  injury  or  death  occurred. 
Mishap  Rate.  Number  of  aviation  mishaps  per  100,000  flight  hours  (DON,  2001). 
Mishap  Severity  Class.  Mishap  severity  classes  are  based  on  personnel  injury  and 
property  damage  (DON,  2001): 

Class  A.  A  mishap  in  which  the  total  cost  of  property  damage  (including 
all  aircraft  damage)  is  $  1 ,000,000  or  greater;  or  a  naval  aircraft  is  destroyed  or 
missing;  or  any  fatality  or  permanent  total  disability  occurs  with  direct 
involvement  of  naval  aircraft. 

Class  B.  A  mishap  in  which  the  total  cost  of  property  damage  (including 
all  aircraft  damage)  is  $200,000  or  more,  but  less  than  $  1 ,000,000  and/or  a 
permanent  partial  disability,  and/or  the  hospitalization  of  five  or  more  personnel. 

Class  C.  A  mishap  in  which  the  total  cost  of  property  damage  (including 
all  aircraft  damage)  is  $20,000  or  more  but  less  then  $200,000  and/or  injury 
results  in  five  or  more  lost  workdays. 

Naval  Aircraft.  Refers  to  US  Navy,  US  Naval  Reserve,  US  Marine  Corps,  and 
US  Marine  Corps  Reserve  aircraft. 
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The  Naval  Aviation  Safety  Program  (OPNAVINST  375Q.6R).  US  Navy 
instruetion  outlining  Naval  Aviation’s  safety  program.  (DON,  2001). 

Operational  Risk  Management  (ORM).  A  deeision  making  tool  to  inerease 
effeetiveness  (and  henee  decrease  accidents)  by  anticipating  hazards,  reducing  the 
potential  for  loss  due  to  these  hazards,  and  thus  increasing  the  probability  of  a  successful 
mission  (DON  1997). 

NATOPS  General  Flight  and  Operating  Instructions  (OPNAVINST  3710).  US 
Navy  instruction  outlining  Naval  Air  Training  and  Operating  Standardization  program  to 
improve  combat  readiness  and  achieving  a  substantial  reduction  in  aircraft  mishaps 
(DON  1997). 

F.  CHAPTER  ORGANIZATION 

Chapter  II  contains  a  literature  review  on  the  development  of  a  prototype  database 
tool  to  identify  human  error  involvement  and  patterns  in  aviation  maintenance  mishaps. 
The  methods  used  in  this  study  are  discussed  in  Chapter  III.  The  results  of  this  study  are 
presented  in  Chapter  IV.  Lastly,  Chapter  V  contains  the  study’s  findings,  conclusions, 
and  recommendations. 
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II.  LITERATURE  REVIEW 


A.  OVERVIEW 

The  literature  studied  encompasses  human  error,  maintenance  error  in  aviation, 
and  error  classification  and  analysis.  It  includes  work  from  textbooks,  research  articles, 
and  master  theses  pertaining  to:  (1)  management  of  accident  information,  (2)  human  error 
theories  and  their  relation  to  maintenance  related  aviation  mishaps,  and  (3)  design  and 
usability  of  a  mishap  database  tool.  This  information  provides  the  foundation  for  the 
ongoing  expansion  and  refinement  of  a  prototype  maintenance  error  analysis  and 
reporting  mishap  database  tool.  While  numerous  efforts  are  ongoing  to  reduce  the 
number  of  Class  A  FMs  in  Naval  Aviation,  there  is  potential  for  further  improvement,  in 
particular  the  area  of  maintenance  error. 

B.  ACCIDENT  INFORMATION 

1.  Investigation 

Grimaldi  &  Simonds  (1984)  detailed  a  four-part  process  for  accident 
investigations.  First  the  investigator  must  explore  the  history,  including  activities 
occurring  both  during  and  prior  to  the  event.  Second,  the  investigator  must  collect  as 
many  facts  relating  to  the  incident  as  possible,  from  reliable  witnesses,  videotapes, 
maintenance  and  training  records.  Next,  the  physical  environment  associated  with  the 
accident  must  be  studied.  Finally,  common  causal  factors  can  be  used  to  determine 
probable  causal  factors  of  the  mishap.  This  process  parallels  aspects  of  that  provided  by 
Diehl  (1991)  in  his  model  of  aviation  accident  investigation. 

DiehFs  (1991)  three-stage  model  of  accident  investigation  and  prevention  focuses 
on  human  performance  and  systems  safety  considerations  (see  Figure  2).  The  Accident 
Generation  stage  involves  hazard  identification.  The  mere  existence  of  hazards  has  the 
potential  to  lead  to  an  incident,  or  worse,  an  accident.  A  study  of  thousands  of  accidents 
by  Heinrich  (1941)  determined  that  for  every  major  accident,  there  are  approximately  30 
minor  accidents,  and  300  hazardous  incidents.  This  relationship  among  hazards, 
incidents,  and  accidents  also  applies  to  aviation  safety  (Diehl,  1991) 
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Figure  2.  Accident  Generation,  Investigation,  and  Prevention  Elements 

(From  Diehl,  1989) 

The  Accident  Investigation  stage  includes  the  collection,  analysis,  and  review  of 
accident  data.  Mishaps  rarely  result  from  a  single,  sudden  event,  but  rather  they  are  tied 
to  a  series  of  events  degrading  equipment  and  crew  performance  until  an  accident  is 
inevitable  (Nance,  1986).  Aircraft  accident  investigation  procedures  require  investigators 
to  determine  what  happened,  and  subsequently  what  caused  the  mishap.  The  analysis  is 
captured  in  a  final  report  containing  accepted  findings,  root  causes,  and  prevention 
recommendations.  This  phase  is  subjective  in  nature,  but  is  based  on  examining 
comparative  data,  on  aircraft  performance,  and  on  human  capabilities/limitations. 

The  Accident  Prevention  stage  details  the  methods  used  to  avoid  future  accidents. 
There  are  four  categories  of  accident-prevention  measures:  (1)  eliminating  hazards  and 
risks,  (2)  incorporating  safety  features  (3)  providing  warning  devices,  and  (4)  establishing 
procedural  safeguards.  As  one  travels  from  right  to  left  along  the  bottom  leg  of  Diehl’s 
triangle  (see  Figure  2),  the  measures  become  less  expensive,  less  effective,  and  less 
restrictive  (Diehl,  1991). 

2.  Reporting 

Accident  reports  generally  centered  on  number  of  episodes  and  observations  per 
unit  of  time  (Brown,  1990).  Frequencies  and  rates,  however,  do  not  provide  a  sound 
basis  for  understanding  accidents.  The  traditional  reporting  format  does  not  normally 
capture  human  factors  information  (Adams,  Barlow,  &  Hiddlestone,  1981).  Ensuring 
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collection,  classification,  and  data  recording  methods  are  accurate  and  reliable  will 
significantly  assist  in  the  determination  of  causes  and  prevention  of  future  mishaps  and 
overall  increase  the  usability  of  the  mishap  report. 

Three  elements  critical  to  ensuring  accurate  and  reliable  mishap  reports  are 
(Chapanis,  1996):  (1)  properly  trained  investigators,  (2)  good  accident  reporting  forms, 
and  (3)  a  centralized  facility  for  dealing  with  reports.  Analyzing  typical  reporting 
systems  data  is  accomplished  through  the  following  process  (Wood,  2000): 

•  collecting  data  on  past  accidents  within  a  population; 

•  dividing  the  sample  into  groups  with  and  without  accidents; 

•  obtaining  measurements  of  individual  characteristics  on  all  participants; 

•  statistically  comparing  the  two  groups;  and 

•  identifying  any  significant  difference  between  the  two  groups,  associating 
the  differential  characteristic  with  accidents. 

These  methods  result  in  a  more  complete  and  thorough  analysis  effort. 

3.  Data  Management 

In  order  for  data  to  be  useful  in  the  prevention  of  accidents,  it  must  be  collected 
and  properly  cataloged  and  stored  for  future  inquires  (National  Safety  News,  1975). 
Coding  the  data  and  the  use  of  databases  to  store  the  information  have  become 
universally  accepted  methods.  In  1975  the  National  Safety  Council  established  a  method 
where  numerical  codes  are  assigned  to  the  different  classifications  in  the  mishap 
(National  Safety  News,  1975).  This  aggregation  of  mishap  data  permits  trend 
identification  and  factor  concentration  to  focus  on  specific  causal  factors.  Obtaining  data 
alone  will  not  prevent  future  mishaps;  the  conditions  contributing  to  the  incident  must  be 
corrected.  Further,  it  can  be  argued  that  not  only  should  accidents  be  analyzed,  but 
“near-miss”  situations  should  be  addresses,  as  well  (Pimbel  &  O’Toole,  1982). 
Recognition  of  near-misses  identify  potential  conditions  or  practices  that  are  accident- 
producing  types  and  prevent  their  future  occurrence. 

Setting  up  a  computer  analysis  program  can  reduce  man-hours  involved  in 
reviewing  mishap  histories  (Kuhlman,  1977).  Computer  analysis  tools  can  significantly 
aid  reviews  of  mishap  histories  (Grimaldi  &  Simonds,  1984).  To  be  a  truly  effective 
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system  in  reducing  future  accidents  and  incidents,  the  tools  must  be  well  organized  and 
tabulate  the  data  logically.  The  user  interface  to  the  data  must  be  presented  sensibly  and 
in  a  easy  to  understand  user-friendly  format. 

4.  Accident  Prevention 

Accident  prevention  began  in  the  first  part  of  the  20th  century  when  employers 
realized  that  it  was  less  expensive  to  prevent  accidents  than  to  pay  for  their  consequences 
(Petersen,  1978).  Initially  accident  prevention  was  based  on  a  notion  that  people 
committing  unsafe  acts,  not  their  working  environment,  were  to  blame  for  most  accidents 
(Heinrich,  1959).  This  accepted  wisdom  fostered  a  preoccupation  with  assigning  blame  to 
people;  a  practice,  which  hindered  the  development  of  systematic  accident  prevention 
well  into  the  latter  half  of  this  century  (Manuele,  1981).  Focusing  on  people  and  not  on 
the  environment  in  which  they  operate  tended  to  obscure  a  subset  of  associated  causal 
factors.  This  is  especially  true  with  systems  that  persistently  expose  individuals  to 
hazards  (Schmidt,  1996).  The  practice  of  blaming  the  individual  and  not  the  environment 
still  exists  even  with  advances  in  accident  prevention  over  the  past  decades.  In  order  to 
prevent  future  accidents,  they  must  be  analyzed  in  terms  of  the  environment  in  which 
they  occur  and  not  point  all  the  blame  to  the  individual. 

Organizations  confronted  with  the  challenge  of  how  best  to  protect  themselves 
and  their  employees  from  accidents  have  two  options:  (1)  insurance,  and  (2)  accident 
prevention  programs  (Pate-Comell,  1996).  Organizations  typically  employ  both  options 
(Kanis  &  Weegels,  1990).  The  most  effective  accident  prevention  strategies  employ 
systems  engineering  (Hawkins,  1993).  Developed  in  the  1950s  the  system  engineering 
approach  was  a  part  of  the  United  States  military’s  large-scale  weapons  programs.  It 
transforms  operational  needs  into  a  description  of  system  parameters  and  integrates  them 
to  optimize  overall  system  effectiveness  (Edwards,  1988).  Systems  engineering  focuses 
the  level  of  analysis  on  the  smallest  identifiable  system  components  and  how  they 
interact  (Bird,  1980).  The  strategy  of  focusing  on  the  system  through  the  development  of 
well-defined  system  components  exposes  information  that  would  have  remained 
unknown  without  a  system-level  evaluation  (Miller,  1988).  System  engineering  not  only 
breaks  down  the  system,  but  also  pays  attention  to  the  strengths  and  limitations  of  the 
human  operator  as  an  integral  part  of  the  system  (Heinrich,  Peterson,  &  Roos,  1980). 
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Numerous  reviews  suggest  that  80  to  90  percent  of  accidents  are  attributable  to  human 
error  (NSC,  2001).  Therefore,  to  totally  understand  way  a  system  failed,  human  factors 
associated  with  the  accident  must  be  evaluated. 

C.  HUMAN  ERROR 

Analyzing  and  correcting  for  human  error  in  aviation  can  greatly  increase  safety. 
There  are  numerous  theoretical  approaches  to  examine  mishaps  involving  human  error 
(Goetsch,  1996).  Some  of  these  approaches  have  their  roots  in  industrial  safety,  while 
others  are  viewed  from  a  more  complex  systems  perspective,  with  an  emphasis  on  human 
factors  and  the  operator.  Three  well  recognized  approaches  are  outlined  in  Table  1. 

Table  1.  Theoretical  Approaches  to  Defining  Accident  Processes  (From  Schmidt, 
1998) 


Source 

Model 

Approach 

Industrial  Safety 

Heinrich’s  Domino  Theory 

Linear 

Systems  Safety 

Edwards’  SHEL  Model 

Interface 

Human  Factors 

Reason’s  Swiss  Cheese  Model 

Vertical 

1.  Heinrich’s  “Domino”  Theory 

Heinrich’s  Domino  Theory  views  accidents  as  a  linear  sequence  of  related  factors 
(see  Figure  3)  or  chain  of  events  that  lead  to  an  actual  mishap  (Bird,  1980).  This  theory 
is  built  upon  two  central  precepts:  (1)  injuries  are  caused  by  the  action  of  preceding 
factors,  and  (2)  removal  of  the  central  factors  negates  the  actions  of  the  preceding  factors, 
and  in  doing  so,  prevents  accidents  and  injuries  (Goetsch,  1996).  Domino  Theory 
encompasses  a  five-step  sequence  (Heinreich,  Petersen  &  Roos,  1980): 

1 .  Lack  of  Control:  This  is  a  management  issue  where  the  emphasis  is  placed  on 
the  control  exercised  in  a  situation  for  an  array  of  factors. 

2.  Basic  Cause(s):  This  identifies  the  origin(s)  of  the  causes  and  includes  aspects 
such  as  human  factors,  environmental  factors,  or  job-related  factors. 

3.  Immediate  Cause(s):  This  includes  substandard  practices  and  conditions  that 
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are  symptoms  of  the  basie  eauses. 

4.  Ineident:  This  typieally  involves  eontaet  with  the  hazard,  and  for  example, 
results  in  a  fall  or  impact  with  moving  objects. 

5.  Person  Injury  and  Property  Damage:  This  includes  lacerations,  fractures, 
death  and  material  damage. 

Much  like  falling  dominos,  each  step  causes  the  next  to  occur.  If  factors  from  any  of  the 
first  three  dominos  are  removed,  the  chain  of  events  will  be  broken  and  the  accident  will 
be  prevented. 


Figure  3.  Domino  Theory  Model  (From  Bird,  1980) 


2.  Edwards’ SHEL  Model 

Developed  in  the  early  1970s,  the  SHEL  Model  provides  an  effective  means  to 
evaluate  human-machine  systems  failures  (Edwards,  1988).  The  model  describes 
systems,  problem  areas,  and  provides  a  framework  for  accident  investigation. 
Furthermore,  the  model  identifies  human-machine  systems  failures  and  classifies  them 
into  four  dimensions:  (1)  Software,  (2)  Hardware,  (3)  Environment  conditions, 
and  (4)  Liveware. 

•  Software:  Typically  a  collection  of  documents  including  rules,  regulations, 
laws,  orders,  standard  operating  procedures,  customs,  practices,  and  habits 
that  govern  how  a  system  operates  and  information  is  organized 

•  Hardware:  Buildings,  vehicles,  equipment  and  materials  of  which  the  system 
is  comprised  and  the  operator  works  with/in. 

•  Environmental  conditions:  Weather,  lighting,  space,  etc. 
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•  Liveware:  People  involved  directly  with/in  and  tied  to  the  system. 

The  SHEL  model  (Edwards,  1988)  is  composed  of  these  four  basic  components 
and  the  interface  between  them  (see  Figure  4).  Failures  may  occur  in  the  system  when 
one  component  or  the  connections  between  them  fail.  Mishaps  are  rarely  associated  with 
only  one  component  or  interface;  they  are  in  fact  caused  by  the  interaction  of  many 
factors  (Shappell  &  Wiegmann,  1997).  This  is  affirmed  in  the  Naval  Aviation  system 
program,  that  is  based  on  necessitarianism  —  mishaps  are  the  inevitable  result  of  their 
antecedent  causes  which  preceded  them  (DON,  2001). 

Software 
Hardware 
Environment 
Liveware 
Liveware 

Figure  4.  SHEL  Model  of  System  Design  (From  Hawkins,  1993) 

3.  Reason’s  “Swiss  Cheese”  Model 

Reason’s  (1990),  Swiss  Cheese  Model  is  an  internationally  accepted  perspective 
on  accident  causation.  It  employs  a  human  factors  approach  to  view  the  vertical 
association  of  a  collection  of  factors  that  eventually  lead  to  an  accident.  The  Swiss 
Cheese  Model  distinguishes  errors  into  two  types:  1)  active  failures  —  the  effects  felt 
immediately,  and  2)  latent  conditions,  where  effects  may  lie  dormant  until  triggered  by 
other  mitigating  factors.  Put  simply,  latent  conditions  set  “the  stage”  for  an  accident 
while  active  failures  are  the  final  catalyst  when  a  mishap  occurs.  Defenses  or  safeguards 
in  a  system  can  prevent  latent  conditions  from  taking  effect,  thus  reducing  the  probability 
for  an  active  failure  to  occur.  The  model  can  be  seen  as  a  row  of  Swiss  cheese  slices, 
each  vertical  slice  representing  a  defense  layer,  and  each  hole  representing  a  failed  or 
missing  defense.  At  times  the  holes  may  be  enlarged  and  can  be  aligned  leading  to  a 
mishap  (see  Figure  5). 
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Figure  5.  Swiss  Cheese  Model  (From  Reason,  1990) 

4.  Human  Factors  Analysis  &  Classification  System  (HFACS) 

In  order  to  capture  human  errors  in  Naval  Aviation  mishaps,  NSC  staff  developed 
the  HFACS  taxonomy  (DON,  2001).  The  goal  of  this  taxonomy  is  to  identify  areas  for 
potential  intervention  by  fully  describing  factors  that  are  precursors  to  accidents.  HFACS 
evolved  from  the  expansion  of  Reason’s  Swiss  Cheese  Model  and  incorporated  features 
of  Heinrich’s  Domino  Theory  and  Edwards’  SHEL  Model  (Shappel  &  Wiegmann,  1997). 

The  resulting  HFACS  taxonomy  focuses  on  aircrew  errors  and  identifies  both 
active  failures  and  latent  conditions  within  four  categories:  1)  Unsafe  Acts,  2)  Pre¬ 
conditions  for  Unsafe  Acts,  3)  Unsafe  Supervision,  and  4)  Organizational  Influences 
(DON,  2001).  NSC  has  adopted  the  HFACS  Model  for  analyzing  human  error  in  Naval 
Aviation  mishaps  and  uses  it  as  a  targeting  tool  for  appropriate  prevention. 

5.  HFACS  -Maintenance  Extension  (HFACS— ME) 

Although  very  useful,  the  HFACS  taxonomy  only  focused  on  aircrew  errors. 
Schmidt,  Schmorrow,  and  Hardee,  (1997)  noted  that  this  taxonomy  could  be  extended  to 
include  maintenance  errors;  hence,  they  developed  HFACS— ME  to  classify  maintenance 
mishaps  causal  factors.  This  new  classification  system  captures  maintenance  human 
factors  by  facilitating  the  recognition  of  absent  or  defective  defenses  at  four  levels: 

(1)  Unsafe  Management  Conditions,  (2)  Unsafe  Maintainer  Conditions,  (3)  Unsafe 
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Working  Conditions  and  (4)  Unsafe  Maintainer  Acts  (see  Figure  6).  This  taxonomy 
visibly  addresses  Marx’s  (1998)  legitimate  concern  that  human  error  has  been  under 
served  by  traditional  maintenance  error  analysis  systems.  Effectively  employed, 
HFACS— ME  is  used  to  identify  maintenance  errors  and  their  causes  and  target 
intervention  strategies  to  avoid  possible  future  mishaps. 


Figure  6.  HFACS -ME  Model  (From  DON,  2001) 

Management,  Maintainer,  and  Working  Conditions  are  latent  conditions  that  can 
impact  the  performance  of  a  maintainer  (Schmidt,  Schmorrow,  &  Hardee,  1997).  These 
latent  conditions  may  contribute  to  an  Unsafe  Maintaner  Act,  an  active  failure,  and 
directly  lead  to  a  Maintenacne  incident,  personal  injury,  or  an  unsafe  maintenance 
condition.  Unsafe  Maintainer  Acts  are  active  failures,  which  directly  or  indirectly  lead  to 
a  latent  condition  that  the  aircrew  will  have  to  deal  with  during  flight.  Maintenance 
conditions  have  the  potential  to  become  a  latent  condition,  which  will  manifest  into  an 
active  failure  in  flight  that  the  aircrew  will  then  have  to  address.  Maintainer  working 
conditions,  as  compared  to  those  of  the  aircrew,  will  often  play  a  more  significant  role  in 
errors  observed  during  maintenance  evolutions  (DON,  2001).  The  three  orders  of 
maintenance  error:  (1)  first,  (2)  second,  and  (3)  third  order,  reflect  a  decomposition  of  the 
error  type  from  a  macro  to  a  micro  perspective  (see  Table  2). 


15 


Table  2.  HFACS-ME  Categories  (From  DON,  2001) 


First  Order 

Second  Order 

Third  Order 

Management  Conditions 

Organizational 

Inadequate  Processes 

Inadequate  Documentation 

Inadequate  Design 

Inadequate  Resources 

Supervisory 

Inadequate  Supervision 

Inappropriate  Operations 

Uncorrected  Problem 

Supervisory  Misconduct 

Maintainer  Conditions 

Medical 

Adverse  Mental  State 

Adverse  Physical  State 

Unsafe  Limitation 

Crew  Coordination 

Inadequate  Communication 

Inadequate  Assertiveness 

Inadequate  Adaptability/Flexibility 

Readiness 

Inadequate  Training/Preparation 

Inadequate  Certification/Qualification 

Personnel  Readiness  Infringement 

Working  Conditions 

Environment 

Inadequate  Lighting/Light 

Unsafe  Weather/Exposure 

Unsafe  Environmental  Hazards 

Equipment 

Damaged/U  nserviced 

Unavailable/Inappropriate 

Dated/Uncertified 

Workspace 

Confining 

Obstructed 

Inaccessible 

Maintainer  Acts 

Error 

Attention/Memory 

Knowledge/Rule 

Skill/Technique 

Judgment/Decision 

Violation 

Routine 

Infraction 

Exceptional 

Flagrant 
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Management  Conditions,  either  organizational  or  supervisory,  may  contribute  to 
an  active  failure  due  to  unforeseen  events  or  lack  of  proper  leadership.  Maintainer 
Conditions  that  can  contribute  to  an  active  failure  include:  (1)  medical,  (2)  crew 
coordination,  and  (3)  readiness  factors  by  the  maintainer.  Working  Conditions  that  can 
contribute  to  an  active  failure  include:  (1)  environment,  (2)  equipment,  and  (3)  the 
workspace  the  maintainer  operates.  (DON,  2001) 

6.  Maintenance  Error  Information  Management  System  (MEIMS) 

Using  the  HFACS— ME  taxonomy  as  a  framework,  a  MEIMS  prototype  database 
tool  was  developed  (Fry,  2000).  MEIMS  is  intended  for  fleet  users  to  collect,  catalog, 
collate,  and  analyze  human  error  in  Naval  Aviation  maintenance  mishaps.  Wood  (2000) 
after  refining  MEIMS,  conducted  a  usability  study  and  demonstrated  the  tool  was 
effective,  yet  lacking  in  some  areas.  Specifically,  MEIMS  was  hard  to  navigate,  required 
HFACS— ME  familiarity/training,  and  had  poor  data  entry  configuration.  Woods  (2000) 
concluded  that  for  MEIMS  to  reach  its  full  potential,  it  needed  design  refinements  and 
more  usability  testing. 

McCracken  (2000)  further  refined  MEIMS  by  developing  a  user  tutorial.  He 
administered  the  tutorial  to  half  of  his  test  subjects.  The  tutorial  group  found  MEIMS  to 
be  of  more  interest  than  the  non-tutorial  group.  Both  groups,  those  given  the  tutorial  and 
those  without,  advocated  MEIMS  relevance  to  maintenance  operations  and  strongly 
endorsed  it,  but  they  also  pointed  out  additional  potential  problem  areas,  to  include: 

(1)  improve  the  graphical  user  interface,  (2)  include  the  tutorial  as  part  of  MEIMS, 

(3)  develop  normal  graph  presentations,  (4)  bring  the  database  up  to  date,  and  (5)  make 
MEIMS  (and  the  tutorial)  available  on  the  World  Wide  Web.  It  was  concluded  that  once 
available  online,  and  with  the  proper  training,  MEIMS  should  be  a  useful  tool  in  the 
effort  to  preserve  lives,  material  and  readiness  (McCracken,  2000). 

D.  TOOL  DESIGN  CONSIDERATIONS 

I.  System  Design 

The  usability  of  any  software  product  can  be  linked  directly  to  its  user  interface 
(Wickens,  Gordon  &  Liu,  1997).  An  easy  to  understand  and  manageable  user  interface 
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will  have  postive  usability  evaluations.  Cleary,  the  user  interface  is  the  most  important 
factor  in  determining  the  success  or  failure  of  a  software  application  (Wickens,  Gordon  & 
Liu,  1997).  To  maximize  the  usability  of  an  interface,  Shneiderman  (1997)  proposed 
eight  golden  rules  of  graphical  user  interface  design:  (1)  strive  for  consistency,  (2)  enable 
frequent  users  to  use  shortcuts,  (3)  offer  informative  feedback,  (4)  design  dialogs  to  yield 
closure,  (5)  offer  error  prevention  and  simple  error  handling,  (6)  permit  easy  reversal  of 
actions,  (7)  support  internal  locus  of  control,  and  (8)  reduced  short-term  memory  load. 
Designing  an  effective  interface  will  increase  the  usability  of  any  program. 

Consistency  is  the  rule  most  frequently  broken  when  designing  a  user  interface. 

In  similar  situations  the  same  action  sequence  should  be  required.  These  consistencies 
include;  identical  terminology,  menus,  and  help  screens  as  well  as  layout,  color, 
capitalization  and  fonts.  Fast  display  rates  and  short  response  time  are  attractions  for 
frequent  users.  To  increase  the  pace  of  interaction,  shortcuts  should  be  provided;  this  can 
be  accomplished  with  abbreviations,  special  keys,  hidden  commands  and  macros.  For 
every  action,  weather  minor  or  major,  there  should  be  varying  degrees  of  information 
feedback  to  the  user.  This  allows  the  user  to  fully  understand  their  current  status.  Every 
event  should  have  a  beginning,  middle  and  end.  Information  feedback  at  the  completion 
of  an  event  will  give  the  user  closure,  a  sense  of  accomplishment  that  the  action  was 
complete.  (Shneiderman,  1997) 

A  system  should  be  designed  such  that  a  user  cannot  make  a  serious  error.  In 
handling  error  the  system  should  detect  the  error  and  provide  simple  and  specific 
instruction  for  recovery.  To  relieve  user  anxiety  and  encourage  exploration  the  system 
should  permit  reversible  actions.  Designing  an  internal  locus  of  control  allows  the  user 
to  feel  in  charge  of  the  system  and  that  the  system  respond  to  their  actions.  This  is 
accomplished  by  making  the  user  the  initiator  of  action  rather  than  the  responder  to 
actions.  Lastly,  to  reduce  short-term  memory,  system  displays  need  to  be  kept  simple, 
multiple  page  displays  to  be  consolidated,  training  time  allotted  for  new  programs  and 
integrated  assistance  information.  (Shneiderman,  1997) 

2.  Usability  Study 

According  to  Nielson  (1998)  the  usefulness  of  a  system  is  determined  by  two 
components:  (1)  the  utility,  and  (2)  usability.  Utility  is  a  measure  of  relevancy;  if  the 
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system  is  irrelevant  to  the  user  it  will  be  a  poor  system  regardless  of  its  design.  Usability 
is  a  measure  of  how  effeetive  the  user  can  navigate  the  system.  Usability  is  the  measure 
of  the  quality  of  the  user  experience  when  interacting  with  a  system,  such  as  a  Web  site, 
software  application,  or  any  user  operated  device.  Nielson  further  breaks  usability  into 
five  characteristics:  (1)  ease  of  learning;  how  fast  can  a  new  user  sufficiently  learn  the 
program,  (2)  efficiency  of  use;  once  the  system  is  learned,  how  fast  the  user  can  complete 
tasks,  (3)  memorability;  how  effective  can  previous  users  accomplish  tasks  without 
relearning  the  system,  (4)  error  frequency  and  severity;  how  many  errors  occurred  and 
how  were  they  recovered,  and  lastly  (5)  subjective  satisfaction;  the  users  approval  with 
the  system.  All  systems  have  all  five  of  the  usability  characteristics  and  all  need  to  be 
considered  in  any  design  project.  (Nielson  1998) 

Frokjaer,  Haertzum  and  Hombaek  (2000)  adopted  the  International 
Standardization  Organization  definition  for  usability,  which  consists  of  three  distinct 
aspects.  First  is  effectiveness,  which  is  the  accuracy  and  completeness  with  which  users 
achieve  certain  goals.  Indicators  of  effectiveness  include  quality  of  solution  and  error 
rates.  Second  is  efficiency,  which  is  the  relation  between  the  accuracy  and  completeness 
with  which  users  achieve  certain  goals  and  the  resources  expanded  in  achieving  them. 
Measures  include  task  completion  time  and  learning  time.  Third  is  satisfaction,  which  is 
the  user’s  comfort  with  and  positive  attitudes  toward  the  use  of  the  system.  A  successful 
software  tool  will  be  effective,  efficient  as  well  as  satisfying  to  the  users  (Frokjaer, 
Haertzum  &  Hombaek,  2000). 

Usability  testing  can  ensure  all  contractual  requirements  have  been  completed, 
help  maximize  the  usability  of  the  system,  and  provide  evidence  of  testing  in  cases  where 
legal  issues  may  arise.  Dependant  on  the  need  to  bring  the  system  to  full  production, 
varying  degrees  of  system  errors  will  be  tolerated  during  testing.  However,  as  the 
number  of  system  inputs  increase,  the  testing  becomes  more  difficult  yet  these  tests  are 
increasingly  needed.  (Shneiderman,  1997) 

Informal  demonstrations  to  colleagues  or  customers  can  provide  useful  feedback, 
but  formal  reviews  by  experts  have  proven  to  supply  more  effective  feedback  (Nielsen  & 
Mack,  1994).  Thus,  system  design  and  test  should  include  expert  reviewers  who  usually 
can  offer  comprehensive  report  on  system  problems  and  make  recommendations.  Expert 
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reviews  can  be  heuristic  evaluations,  guidelines  reviews,  consistency  inspections, 
cognitive  walkthroughs,  or  formal  usability  inspections  (see  Table  3).  Expert  reviews  can 
be  scheduled  at  project  milestones  in  the  development,  when  experts  are  available,  or 
when  the  project  team  is  ready. 


Table  3:  Expert  Review  Methods  (From  Shneiderman,  1997) 


Expert-Review 

Method 

Description 

Heuristic  evaluation 

Expert  reviewers  critique  an  interface  to  determine  conformance 

with  a  short  list  of  design  heuristics  such  as  the  eight  golden  rules. 

Guideline  review 

The  interface  is  checked  for  conformance  with  the  organizational 

or  other  guidelines  document. 

Consistency 

inspection 

Experts  verify  consistency  across  a  family  of  interfaces, 

terminology,  color,  layout,  input/output  formats,  within  the 

interfaces,  in  the  training  materials  and  online  help. 

Cognitive- 

walkthrough 

Experts  simulate  users  walking  through  the  interface  to  carry  out 

typical  tasks.  Simulating  the  day  in  the  life  of  the  user  should  be 

part  of  the  evaluation. 

Formal  usability 

inspection 

Experts  hold  courtroom-style  meeting,  with  a  moderator  to  judge, 

to  present  the  interface  and  to  discuss  its  merits  and  weaknesses. 

When  planning  to  conduct  a  usability  study  an  important  consideration  is  the 
length  of  the  study  (Dumas  &  Redish,  1994).  If  the  study  is  integrated  with  the  design 
process,  then  the  length  can  be  reduced  to  a  manageable  level  so  as  to  not  impose  a 
burden  on  the  expert  and  still  provide  useful  feedback.  Formal  testing,  with 
comprehensive  test  reports  requires  eight  to  twelve  weeks,  if  a  strong  collaboration  is 
exhibited  among  team  members  and  a  shortened  report  format  is  used,  then  four  to  six 
weeks  are  required.  If  a  particular  part  of  the  system  is  to  be  studied  with  well- 
established  procedures,  then  one  week  may  be  appropriate.  Although  discouraged,  just- 
in-time  testing  can  provide  useful  information  in  a  few  days,  if  necessary.  The  length  of 
the  study  is  also  a  critical  part  of  the  usability  study  and  can  impact  results. 
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3.  Computer-Human  Interaction  Design  Issues 

Human  Factors  Engineering  existed  long  before  computers  were  developed  and 
examines  the  relationship  between  humans  and  all  types  of  maehines  (Carey,  1991). 

With  the  advent  of  eomputer  systems,  Human  Factors  in  Information  Systems  (HFIS) 
became  an  area  of  interest.  Aeeording  to  Carey  (1991)  several  disciplines  eontribute  to 
and  overlap  with  HFIS:  (I)  Computer  Scienee,  (2)  Management  Information  Systems,  (3) 
Human  Factors  Engineering,  and  (4)  Computer-Human  Interaction.  Each  of  these  areas 
of  study  has  a  focus,  which  overlap  with  HFIS.  Carey  (1991)  uses  a  VENN  diagram  to 
illustrate  the  differences  and  similarities  between  the  diseip lines  and  the  nature  of  HFIS, 
(see  Figure  7).  A  eircle  represents  eaeh  of  the  four  disciplines;  the  intersection  of  the 
cireles  represents  HFIS.  The  portion  of  eaeh  eircle  that  overlaps  with  HFIS  indieates 
how  elosely  each  area  is  related  to  HFIS. 


Figure  7.  Disciplines  Contributing  Knowledge  to 
Human  Factors  in  Information  Systems  (From  Carey,  1991). 

The  following  statements  reflect  the  goals  and  purposes  of  eaeh  of  the  diseiplines:  (1) 
Computer  Scienee  is  about  optimizing  computer  effieiency,  (2)  Management  Information 
Systems  is  about  maximizing  organizational  effeetiveness  through  information, 

(3)  Human  Factors  Engineering  is  about  increasing  system  performanee  by  reducing 
human  error,  (4)  Computer-Human  Interaetion  is  about  increasing  user  effeetiveness  by 
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enhancing  the  user  interface,  and  finally  (5)  HFIS  is  about  increasing  user  effectiveness 
within  an  organization  by  enhancing  the  user  interface  and  other  human-to-computer 
contact  such  as  training  and  user  involvement  in  development  (Carey,  1991). 

Brown  (1989)  states  that  a  useful  design  philosophy  for  developing  user-oriented 
human-computer  interfaces  is  to  consider  the  computer  as  a  tool  to  aid  the  user  in 
accomplishing  tasks.  A  tool,  which  requires  more  time,  training  and  effort  to  use  than  the 
task  requires  without  the  tool,  is  a  poorly  designed  system.  Brown  recommends  an  eight- 
step  strategy  for  developing  an  effective  computer-human  interface:  (1)  establish  the 
human-computer  interface  role  in  system  development,  (2)  know  the  users,  (3)  define  the 
tasks,  (4)  incorporate  design  guidelines,  (5)  train  software  designers  in  computer-human 
interface  design,  (6)  develop  user  interface  software  tools,  (7)  prototype  and  user  testing, 
and  lastly  (8)  designing  by  interactive  refinement.  “The  most  important  thing  to  know 
regarding  your  user  is  that  he  is  not  interested  in  using  your  product.  He  is  interested  in 
doing  his  work,  and  your  product  must  help  him  do  it  more  easily”  (Heckel,  1994).  A 
well-designed  computer  interface  will  aid  users  in  completing  assigned  tasks. 

In  establishing  the  computer-human  interface  role  in  systems  development, 
management  support,  participation  as  team  members,  and  appreciation  for  design 
tradeoffs  are  critical  to  the  success  of  the  design.  Users  must  be  directly  involved  in  the 
design  process  from  its  infancy,  knowing  the  users  are  critical  to  successful  system 
design.  Design  must  be  based  on  an  understanding  of  the  tasks  the  users  will  perform 
with  the  system,  and  the  environment  in  which  it  will  be  used.  User  interface  guidelines 
must  be  developed,  documented,  revised,  maintained  and  customized  to  the  context  and 
constraints  of  the  project.  Systems  analysts,  programmers,  and  other  developers  often 
need  to  be  trained  to  the  concepts  and  philosophy  of  computer-human  interface  design. 
Interface  software  tools  can  enhance  the  consistency  in  the  interface  and  provide  an 
environment  where  interactive  design  is  simple.  It  also  benefits  in  program  modularity, 
data  independence,  and  development  time  and  cost.  Testing  early  in  the  development 
cycle  can  reveal  flaws  in  concepts  or  assumptions  about  what  users  need  and  want. 

Design  features  and  concepts  must  be  tested  on  people  from  the  population  of  users  for 
which  the  system  is  targeted.  Finally,  as  ongoing  tests  reveal  needed  changes  and 
refinements  the  design  must  be  updated.  Early  testing  of  the  design  will  prove  to  keep 


22 


updating  costs  low.  The  problems  discovered  in  a  test  cycle  must  be  resolved  in  a  revised 
design.  Then  to  assure  the  flaws  are  corrected,  the  revised  design  must  be  tested.  (Brown, 
1989) 

E.  SUMMARY 

Over  the  last  half  century,  Naval  Aviation  has  made  significant  strides  in  reducing 
the  mishap  rate.  This  reduction  can  be  attributed  to  standardized  reporting  and 
investigation  system,  the  use  of  systems  engineering,  the  application  of  human  error 
causation  theory  on  mishap  cause  factors.  Mishap  data  is  currently  being  collected, 
cataloged  and  stored  for  future  reference.  This  data  is  useless  in  preventing  future 
fatalities  unless  a  well-defined,  systematic  accident  investigation,  analysis,  and  reporting 
process  is  developed  and  tested.  No  single  universal  system  currently  exits  (Marx,  1998). 
Current  technology  exists  that  make  organization  of  data  relatively  simple  for  the 
program  and  the  user.  For  Naval  Aviation  to  further  increase  its  safety  record,  a  robust 
system  for  analyzing  stored  data  for  trend  analysis  and  prevention  is  needed. 

HFACS  was  developed  to  provide  a  more  useful  data  analysis  system.  With  its 
success,  HFACS  was  expanded  to  cover  maintenance  error,  and  HFACS— ME  was 
developed.  Both  have  proven  to  be  effective  in  capturing  the  nature  of,  and  relationship 
among,  latent  conditions  and  active  failure.  MEIMS  was  developed  as  a  tool  to  capture 
maintenance  error  trends  in  aviation  maintenance  using  the  HFACS— ME  taxonomy.  In 
limited  usability  studies,  MEIMS  proved  to  have  great  potential  to  increase  safety 
awareness.  However,  to  fully  reach  its  potential,  MEIMS  must  be  proven  to  be  a  user- 
friendly  system  that  users  embrace.  In  order  to  achieve  this  goal,  MEIMS  is  undergoing 
further  systems  development  and  more  rigorous  usability  testing.  With  proper  advances, 
MEIMS  has  the  potential  to  analyze  data,  find  trends,  save  taxpayers  money,  and 
ultimately  save  lives. 
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III.  METHODOLOGY 


A.  MEIMS  APPLICATION  DEVELOPMENT 

The  original  version  of  MEIMS  was  programmed  in  Mierosoft  Access  97  using 
Visual  Basic  for  Applications  (VBA).  This  prototype  had  very  little  documentation.  The 
FAA  required  that  it  be  upgraded  to  Access  2000.  Because  of  the  lack  compatibility 
between  Access  97  and  Access  2000  a  completely  new  version  of  MEIMS  had  to  be 
developed. 

Flanders  and  Tufts,  two  Computer  Science  graduate  students  at  the  Naval 
Postgraduate  School,  developed  a  revised  version  of  MEIMS  prototype  for  their  research 
using  Access  2000.  The  authors’  contribution  to  the  new  MEIMS  application  was 
requirements  generation,  functional  assessment,  and  program  development  support.  The 
author  also  developed  a  decision  support  investigation  module  based  on  the  HFACS— ME 
taxonomy.  The  investigation  module  assists  the  user  in  determining  mishap  causal 
factors  and  builds  a  preliminary  investigation  report.  A  detailed  description  of  the 
MEIMS  application  prototype  can  be  found  in  Flanders  and  Tufts  (2001)  -  Software  Re¬ 
engineering  of  the  Human  Factors  Analysis  and  Classification  System  -  (Maintenance 
Extension)  Using  Object  Oriented  Methods  in  a  Microsoft  Environment. 

B.  RESEARCH  APPROACH 

The  MEIMS  prototype  tool  was  circulated  to  a  representative  sample  of 
prospective  end-users.  Participants  were  provided  a  prepared  task  list  that  required  them 
to  enter  fictitious  mishap  data  information,  as  well  as  navigate  through  and  utilize 
features  of  the  tool.  The  participants  navigated  through  the  entire  system  and  completed 
an  exit  survey.  The  exit  survey  instrument  included  demographic  background 
information,  as  well  as  quantitative  and  qualitative  survey  items  designed  to  elicit  users’ 
views  and  ideas.  The  resulting  data  was  inserted  into  a  Microsoft  Excel  spreadsheet  for 
analysis.  Content  analysis  was  conducted  on  the  qualitative  survey  questions. 
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C.  DATA  COLLECTION 


1.  Participants 

Fifteen  students  attending  either  the  Naval  Postgraduate  School  (NPS),  or  the 
Aviation  Safety  Office  (ASO)  course,  within  the  School  of  Aviation  Safety  at  NPS, 
severed  as  participants  in  the  research.  The  NPS  is  comprised  of  officers  from  the  Army, 
Navy,  Air  Force,  Marine  Corps,  and  Coast  Guard,  as  well  as  several  foreign  Services. 
Student  demographics  represent  a  wide  cross  section  of  Naval  Aviators,  Naval  Flight 
Officers,  DOD  officers.  Flight  Surgeons,  Aeromedical  Safety  Officers,  and  foreign 
nationals  from  a  variety  aircraft  communities  and  platforms.  ASO  course  graduates  are 
responsible  for  implementing  and  managing  squadron  safety  programs  and  mishap 
investigation  and  reporting.  NPS  graduates,  who  are  designated  aircrew,  typically  return 
to  aviation  units  in  department  head  positions  and  play  an  integral  role  in  safety 
initiatives. 

2.  Apparatus 

Participants  were  introduced  to  the  HFACS— ME  taxonomy  and  the  MEIMS 
prototype  mishap  database  tool  through  the  use  of  a  multimedia  demonstration. 
Participants  had  access  to  three  computer  laboratories  at  the  School  of  Aviation  Safety 
via  login  ID  and  password  to  a  group  account.  Each  computer  system  in  the  labs  is  a 
Pentium  I,  with  a  Windows  2000  operating  system,  and  15-inch  monitor  of  800  x  600 
resolution  (or  better).  All  systems  had  a  fully  functioning  MEIMS  prototype  mishap 
database  tool  loaded  onto  the  hard  drive.  After  gaining  access  to  the  computer,  the 
“MEIMS  Tool”  icon  was  selected  to  open  the  MEIMS  prototype  (see  Appendix  A). 

3.  Instrument 

The  author  constructed  a  participant  usability  survey  consisting  of  three  parts: 

(1)  Participant  demographics,  (2)  Likert-type  quantitative  assessment  statements  ,  and  (3) 
Open  ended  qualitative  items.  Demographic  information  was  collected  by  participants 
selecting  from  a  list  of  descriptors  (rank,  branch  of  Service,  number  of  years  in  Service, 
and  total  flight  hours).  Survey  questions  were  designed  to  determine  if  the  prototype  tool 
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meets  user  investigation,  reporting,  and  analysis  requirements.  Likert-type  statements 
used  a  five-point  rating  scale:  Strongly  Agree,  Agree,  Neutral,  Disagree,  and  Strongly 
Disagree,  to  capture  the  participant’s  opinions.  The  open  ended  questions  were 
constructed  to  elicit  the  subjects  overall  impression  of  the  prototype  software  tool, 
recommendations  for  improvement,  and  an  area  for  additional  comments  not  adequately 
covered  by  the  first  two  portions. 

4.  Procedure 

The  MEIMS  application  containing  a  database  derived  from  compiled  NSC 
maintenance  mishaps  was  loaded  on  computer  systems  in  the  labs  with  24-hour 
accessibility.  There  was  a  MEIMS  icon  on  each  computer  desktop  to  allow  easy  access 
to  the  application.  Before  logging  on,  participants  were  given  a  training  tutorial  and 
instruction  manual  for  MEIMS. 

Testing  was  conducted  over  a  one  week  period.  All  participants  were  given  a 
group  orientation  on  the  purpose,  goals,  and  procedures  for  the  prototype  including  a 
computer  demonstration,  and  materials  necessary  to  carry  out  the  user  test.  These 
materials  included: 

•  Instructions  for  accessing  the  Prototype  tool  -  information  to  log  on  and 
open  the  prototype  (See  Appendix  B). 

•  MEIMS  Evaluation  -  A  series  of  planned  navigation  routes  for  every  area 
of  the  prototype.  (See  Appendix  B). 

•  MEIMS  Exit  Survey  -  Participants  completed  an  exit  survey  composed  of 
demographic  background  questions,  impressions  of  MEIMS  and  the 
investigation  module  and  open  ended  questions  requesting  participant’s 
opinions  (See  Appendix  C). 

All  participants  performed  the  following  actions:  (1)  accessed  the  prototype  tool,  (2) 
navigated  the  system  using  the  prototype  task  list,  and  (3)  provided  feedback  on  the 
system  by  completing  the  exit  survey.  The  anonymously  completed  exit  surveys  were  all 
submitted  through  a  drop  box  provided  in  a  common  area. 
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D.  DATA  TABULATION 


The  collected  data  were  transcribed  from  the  survey  into  a  Microsoft  Excel  2000 
spreadsheet.  The  Likert-type  statements,  based  on  a  five-point  scale,  are  coded  into  the 
software  using  number  1  through  5  to  correspond  respectively  with  (1)  Strongly  Agree, 
(2)  Agree,  (3)  Neutral,  (4)  Disagree,  and  (5)  Strongly  Disagree. 

E.  DATA  ANALYSIS 

Descriptive  statistics  were  generated  using  functions  within  Excel  to  include:  (1) 
mean,  (2)  standard  deviation,  (3)  range,  and  (4)  frequency  distribution  of  the  collected 
data.  Content  analysis  was  conducted  on  the  responses  provided  from  the  open-ended 
survey  questions.  The  data  were  used  to  assess  the  viability  of  the  MEIMS  prototype. 
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IV.  RESULTS 


A.  TEST  SAMPLE 

A  usability  test  was  administered  to  15  students  at  the  Naval  Postgraduate  School. 
All  participants  were  designated  Naval  Aviators  or  Naval  Flight  Officers  and  represented 
a  cross  section  of  the  aviation  commands  that  make  up  the  squadrons  in  the  Navy  and 
Marine  Corps.  No  foreign  Service  students  participated  in  the  survey.  Each  participant 
was  given  a  MEIMS  tool  evaluation  package  with  1 0  tasks  to  complete  and  a  brief 
tutorial  on  HFACS— ME.  After  completing  the  tasks,  participants  were  asked  to  complete 
an  exit  survey.  The  survey  consisted  of  demographic  information  and  queries  regarding 
their  satisfaction  with  MEIMS. 


B.  TEST  TASKS 


The  tasks  were  designed  to  introduce  the  participants  to  MEIMS  and  exercise 
some  of  its  capabilities.  The  tasks  required  the  participant  to  access  all  functional  areas 
of  MEIMS.  Test  performance  is  summarized  in  Table  4. 


Table  4:  Test  Task  Performance 


TASK 

n 

NUMBER  CORRECT 

PERCENTAGE 

1  -  Access  Program 

15 

15 

100% 

2  -  Main  Menu  Opinion 

15 

11 

73% 

3  -  Aircraft  Query 

15 

15 

100% 

4  -  ID  Factors 

15 

15 

100% 

5  -  Access  Information 

15 

14 

93% 

6  -  Access  Information 

15 

13 

87% 

7  -  Access  Information 

15 

14 

93% 

8  -  Graph  Information 

15 

15 

93% 

9  -  ID  Mishap 

15 

15 

100% 

10  -  Investigation  Opinion 

15 

13 

87% 
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The  first  task  was  accessing  the  program;  all  participants  (n=15;  100%)  were  able 
to  access  MEIMS  without  difficulty.  The  second  task  requested  the  participants’  opinion 
of  the  main  menu  (See  Figure  Al,  Appendix  A).  The  majority  of  participants  (n=  11; 
73%)  responded  that  the  menu  was  easy  to  understand.  Other  participants  noted  that  the 
average  computer  user  might  not  understand  “query”.  Tool  tips  would  help  in 
understanding  what  each  function  does,  and  one  participant  commented  that  he  thought 
there  would  be  a  number  of  graphs  under  the  graph  menu.  One  participant  did  not 
answer  this  question.  The  third  and  forth  tasks  required  the  participant  to  query  a  type  of 
aircraft,  from  the  multiple  criteria  menu  (See  Figure  A3,  Appendix  A),  and  then 
determine  how  many  mishaps  exist  in  the  database  for  that  type  aircraft  and  how  many 
factors  existed  for  the  first  mishap.  All  participants  (n=15;  100%)  were  able  to  correctly 
accomplish  these  tasks.  The  fifth,  sixth  and  seventh  tasks  required  the  user  to  draw 
mishap  information  from  the  HFACS— ME  Summary  Form  (See  Figure  A5,  Appendix  A) 
regarding  the  total  number  of  mishaps  and  factors  within  the  database.  Fourteen  (93%) 
were  able  to  correctly  identify  the  answers  for  the  fifth  and  seventh  tasks.  On  the  sixth 
task  thirteen  (86%)  were  able  to  correctly  identify  the  answer.  One  participant  had  the 
right  answers  down  for  his  specific  aircraft,  but  not  the  total  number  of  mishaps  in  the 
database.  The  eighth  task  required  the  user  to  properly  graph  (See  Figures  A6  &  A7, 
Appendix  A)  the  mishap  types  versus  the  organization  and  determine  how  many  aviation 
ground  mishaps  the  U.S.  Navy  has  had  by  clicking  on  a  portion  of  the  graph.  All 
participants  (n=15;  100%)  correctly  identified  the  value.  Several  participants  commented 
on  the  difficulty  of  having  to  click  on  just  the  right  spot  on  the  graph  and  that  some  sort 
of  roll  over  value  would  greatly  enhance  this  function.  The  ninth  task  required  users  to 
open  up  a  specific  report  from  the  report  menu  (See  Figure  A8,  Appendix  A)  and  identify 
the  number  of  total  number  of  Class  B  mishaps.  All  participants  (n=15;  100%)  were  able 
to  correctly  identify  the  value.  The  last  task  requested  the  participants’  opinions  of  the 
investigation  portion  (See  Figures  AlO  to  A20,  Appendix  A)  and  whether  it  helped  them 
identifying  the  causal  factors  associated  with  the  mishap  scenario.  Thirteen  participants 
(87%)  felt  that  the  section  helped  them  in  identifying  the  causal  factors.  Two  participants 
(13%)  did  not  answer  the  question. 
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C.  DEMOGRAPHIC  INFORMATION 


The  data  collected  in  part  one  of  the  exit  survey  consisted  of  demographic 
information  concerning  participant’s  computer  and  aviation  experience  levels.  This 
information  was  used  to  determine  if  the  participants’  level  of  experience  effected  their 
satisfaction  with  the  MEIMS  prototype  database  tool.  Demographic  information  is 
summarized  in  Table  5. 


Table  5.  Demographic  Information 


Demographic 

n 

Number  of  Participants 

Percent 

Squadron  level  maintenance 

15 

15 

100% 

2  +  years  computer  experience 

15 

15 

100% 

Use  Microsoft  Office 

15 

15 

100% 

Use  word  processing  programs 

15 

15 

100% 

Use  word  spreadsheet  programs 

15 

15 

100% 

Use  presentations  programs 

15 

15 

100% 

Use  graphic  related  software 

15 

10 

66% 

Use  E-Mail 

15 

15 

100% 

Use  Database  programs 

15 

10 

66% 

Use  Windows  (3.1-2000) 

15 

15 

100% 

Use  Windows  NT 

15 

11 

73% 

Use  Macintosh 

15 

1 

6% 

Use  Linux 

15 

2 

14% 

Use  Unix 

15 

3 

20% 

Question  one  determined  that  all  participants  had  been  or  are  members  of  aviation 
units  that  performed  squadron  level  maintenance  (n=15;  100%).  Question  two  indicated 
that  all  participants  had  at  least  two  years  of  experience  using  a  computer  (n=15;  100%). 
Question  three  determined  that  all  participants  (n  =  15;  100%)  were  users  of  Microsoft 
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Office.  Question  four  established  a  partieipant’s  familiarity  with  different  software 
applications;  all  participants  (n=15;  100%)  were  familiar  with  processing,  spreadsheet, 
presentation,  and  e-mail.  A  third  (n=10;  66%)  of  the  partieipants  were  familiar  with  both 
graphie  and  database  applieations.  Question  five  revealed  what  operating  system  the 
partieipates  are  familiar  with  working  with.  All  partieipants  (n  =  15;  100%)  are  familiar 
with  Window  3.1,  95,  98,  or  2000  while  several  (n=l  1;  73%)  were  familiar  with 
Windows  NT,  one  (6%)  worked  with  Maeintosh  and  few  (n=3;  20%)  and  (n=2;  14%) 
were  familiar  with  Unix  and  Linux  respectively. 

D.  PARTICIPANT  SATISFACTION  WITH  MEIMS  TOOL 

1.  Responses  to  Impressions  of  MEIMS 

The  information  gathered  in  Part  II  of  the  exit  survey  requested  the  participants’ 
impressions  of  the  MEIMS  tool  and  its  value  to  Naval  Aviation.  Partieipants  responded 
to  five  statements  using  Likert  type  responses  seleeting  from  one  of  five  responses:  (1) 
strongly  agree,  (2)  agree,  (3)  neutral,  (4)  disagree,  and  (5)  strongly  disagree.  Values  of 
five  through  one  respeetively  were  assigned  to  the  statements.  Additionally,  partieipants 
could  make  subjeetive  eomments  on  any  of  the  statements. 

Statement  one  asked  whether  or  not  a  participant  found  MEIMS  to  be  presented  in 
a  logieal  form.  Almost  all  of  the  partieipants  either  strongly  agreed  (n=7;  47%)  or  agreed 
(n=7;  47%)  that  the  MEIMS  prototype  was  in  a  logical  form  while  only  one  (6%)  was 
neutral  on  whether  it  was  logieal.  Figure  8  depiets  the  results  for  “being  in  a  logical 
form”. 
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Figure  8.  MEIMS  is  “In  a  Logical  Form” 

Statement  two  asked  about  the  ease  of  navigation  of  the  prototype.  Almost  all  of 
the  participants  strongly  agreed  (n=8,  54%)  or  agreed  (n=6,  40%)  that  the  MEIMS 
prototype  tool  was  easy  to  navigate.  Only  one  (6%)  was  neutral  on  the  ease  of 
navigation.  The  results  for  easy  navigation  are  depicted  in  Figure  9. 


Figure  9.  MEIMS  is  “Easy  to  Navigate” 

Statement  three  asked  the  participants  if  MEIMS  was  interesting.  Almost  all  of 
the  participants  strongly  agreed  (n=7;  47%)  or  agreed  (n=6;  40%)  that  MEIMS  was 
interesting.  Two  (13%)  of  the  participants  were  neutral  on  the  whether  MEIMS  was 
interesting.  The  results  for  being  very  interesting  are  depicted  in  figure  10. 
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13% 


□  Strongly  Agree  □  Agree  □  Neutral  □  Disagree  ■  Strongly  Disagree 


Figure  10.  MEIMS  is  “Interesting” 

Statement  four  asked  about  the  relevance  of  MEIMS  to  aviation  maintenance 
operations.  All  of  the  participants  either  strongly  agreed  (n=8;  53%)  or  agreed  (n=7, 
47%)  that  MEIMS  is  relevant  to  maintenance  operations.  The  results  for  maintenance 
operations  relevance  are  depicted  in  figure  1 1 . 


Figure  11.  MEIMS  is  “Relevant  to  Maintenance  Operations” 

Statement  five  asked  whether  prototype  concept  was  a  good  one.  All  the 
participants  either  strongly  agreed  (n=13;  87%)  or  agreed  (n=2;  13%)  that  the  MEIMS 
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concept  is  a  good  one.  Figure  12  depiets  the  result  for  the  coneept  goodness. 


Statement  six  asked  whether  the  partieipants  found  the  investigation  tool  helpful. 
All  of  the  partieipants  either  strongly  agreed  (n=l  1;  74%)  or  agreed  (n=4;  26%)  that  the 
investigation  tool  was  helpful.  The  results  are  depicted  in  Figure  13. 


Figure  13:  Investigation  “Tool  is  Helpful” 
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2.  Responses  to  Open-ended  Questions 

Part  III  of  the  exit  survey  eontained  four  open-ended  questions  regarding  the 
partieipants  overall  satisfaetion  with  MEIMS.  All  participants  took  the  opportunity  to 
make  comments.  The  majority  of  comments  were  positive  and  indicated  that  MEIMS 
was  a  good  tool  that  has  the  potential  to  be  extremely  valuable  instrument  in  the 
prevention  of  mishaps 

Question  one  asked  the  participant  to  list  the  most  positive  aspects  of  the 
prototype.  Overall,  the  response  was  positive.  Six  participants  commented  on  MEIMS 
ease  of  use  and  ease  of  navigation.  Five  participants  commented  on  layout  of  MEIMS, 
the  quantity  of  information  and  the  thoroughness.  Several  commented  on  the  features 
that  the  user  has  the  ability  to  capture  trends.  Some  sample  inputs  include: 

•  "^Quantifies  other  factors  of  a  mishap  beyond  the  aircrew.  ” 

•  “It  helps  the  ASO  identify  factors  they  may  not  have  thought  of  ” 

•  “Allows  quick  and  easy  input  of  data  by  trained  Safety  Ojficer /Individual.  ” 

•  “Ability  for  maintenance  supervisors  to  query  for  specific  3’'^  level  factors  for 
briefing  and  training  maintance  personnel.  ’’ 

•  “The  fact  that  it  brings  together  all  the  mishap  data  in  one  place  for  easy 
access.  ” 

Question  two  asked  for  the  most  negative  aspects  of  the  prototype.  Most 
comments  were  problems  or  suggested  improvement  to  the  interface.  A  few  left  this  area 
blank  and  some  comments  made  had  merit,  but  were  beyond  the  scope  of  this  study. 
Several  of  the  comments  focused  on  the  users  lack  of  understanding  with  the  HFACS— 
ME  taxonomy  terms  and  the  fact  that  potential  users  would  need  some  training.  Some 
sample  comments  include: 

•  “A  little  cumbersome  to  use  at  first.  ” 

•  “An  explanation  of  the  fy,  2"^,  and  level  factors  may  be  helpful.  Often  the 

factors  seem  so  similar  that  it  almost  get  confusing  as  to  which  one  is  most 
appropriate.  If  there  was  possibly  something  to  go  back  to  see  some  key 
words  then  it  might  help.  ” 

•  “No  interface  for  retrieving  NSC  data.  ” 
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•  “Learning  curve  once  past  that  everything  is  great.  ” 

Question  three  requested  suggestions  for  changes  to  MEIMS.  For  the  most  part 
participants  reiterated  what  they  had  stated  previously  and  centered  on  training  or  their 
lack  of  knowledge  of  HF ACS— ME.  Several  participants  left  this  area  blank.  A  few  of  the 
comments  include; 

•  “Add  a  few  standard  icons  to  selection  Menu.  ” 

•  “Add  Help  Menu  ” 

•  “Add  tool  tips.  ” 

•  “Online  tutorial  to  get  started.  ” 

•  “Spend  the  time  to  make  it  a  little  more  user  friendly  without  the  aid  of  tool 
evaluation  package,  it  would  be  difficult  to  use.'' 

Question  four  requested  suggestions  for  changing  the  investigation  section  of 
MEIMS.  Some  participants’  comments  again  focused  on  their  lack  of  understanding  with 
the  HFACS— ME  taxonomy  and  the  definitions  of  all  its  terms.  Several  participants  had 
no  comment.  Two  participants  suggested  that  the  user  should  have  the  option  to  go 
through  the  factor  input  wizard  or  manually  enter  the  factors.  They  felt  that  once  the  user 
became  familiar  with  the  taxonomy  the  factor  input  wizard  would  only  slow  them  down. 
Several  others  responded  positively  about  the  ability  to  import  the  preliminary  report  into 
a  Microsoft  Word  document.  Some  comments  include: 

•  “I  can ’t  say  I  would  add  anything  more  to  this  one  particular  portion,  it 
certainly  gives  the  database  a  lot  more  functionality.  ’’ 

•  “Investigation  module  portion  helpful  but  not  as  a  stand-alone  application,  it 
would  need  to  be  integrated.  ” 

•  “Possible  examples  of  descriptions.  ” 

•  “Add  help  menu  that  incorporate  the  legal  and  layman ’s  descriptions  of 
mishap  categories.  ” 

•  “Should  be  able  to  skip  the  factor  input  status  if factors  are  known.  ” 

Lastly  there  was  a  space  for  additional  comments,  they  include; 

•  ''Great  for  a  safety  standown  to  look  at  the  trends  instead  of  always  glossing 
over  on  a  case  by  case  basis.  ’’ 
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•  “Good  addition  to  the  ASO  tool  kit.  ” 

•  “Good  training  to  give  the  maintenance  department  -  so  they  know.  ” 

•  “Good  to  go!” 
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V.  SUMMARY,  CONCLUSIONS,  AND  RECOMMENDATIONS 


A.  SUMMARY 

Naval  Aviation  has  always  had  great  demand  for  high  levels  of  operational 
readiness,  equipment  availability,  and  personal  training.  These  trends  continue  today, 
despite  reduced  budgets  and  aging  aircraft.  To  continue  to  meet  this  operational 
demand,  it  become  vitally  crucial  to  protect  our  assets,  as  replacement  airframes  are  not 
likely  in  the  foreseeable  future.  In  this  financially  strapped  environment,  costs  of  Naval 
Aviation  mishaps,  in  terms  of  operational  readiness,  material  resources,  and  operational 
capability  are  too  high.  Therefore,  reducing  the  number  of  mishaps  is  crucial  to  the 
continuing  success  of  Naval  Aviation.  Although  not  all  mishaps  are  avoidable,  reducing 
the  mishaps  involving  human  error  becomes  imperative. 

In  an  effort  to  reduce  human  factor  errors  in  Naval  Aviation  mishaps,  recent 
efforts  have  targeted  aircrew  error.  This  emphasis  has  resulted  in  the  comprehensive 
HFACS  taxonomy.  These  early  efforts  resulted  in  a  reduction  of  mishaps,  but  not  to  the 
extent  Naval  Aviation  had  hoped  to  achieve.  It  was  soon  realized  that  the  scope  of 
mishap  prevention  must  be  expanded  beyond  aircrew  error  to  included  maintainer  errors. 
This  expanded  scope  resulted  in  the  HFACS  Maintenance  Extension,  or  HFACS— ME. 
This  taxonomy  proved  to  be  an  acceptable  method  for  classifying  maintainer  errors. 
Using  the  HFACS— ME  taxonomy,  a  clear  picture  of  the  human  factor  errors  contributing 
to  maintenance  mishaps  could  be  formulated. 

To  fully  tap  into  the  potential  of  the  HFACS— ME  taxonomy,  an  existing 
prototype  database  tool,  Maintenance  Error  Information  Management  System  (MEIMS) 
was  developed  and  refined.  This  enhanced  MEIMS  prototype  mishap  database  tool, 
based  on  the  HFACS— ME  taxonomy,  is  a  safety  information  management  system  used  to 
facilitate  the  characterization  and  analysis  of  human  error  in  Naval  Aviation  maintenance 
mishaps.  Users  are  able  to  query  data,  custom  make  graphs,  view  reports,  conduct  a 
preliminary  investigation,  as  well  as  download  and  updated  database  file.  A  Tool  such 
as  MEIMS  has  the  potential  to  identify  human  errors  patterns  or  trends  and  assist  safety 
personnel  in  intervention  development. 
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B.  CONCLUSIONS 


The  participants’  high  level  of  satisfaction  with  the  MEIMS  prototype  mishap 
database  tool  indicated  a  need  for  quick,  accurate  mishap  data  information  for  use  in 
training,  analysis,  and  investigations.  Participant  feedback  demonstrated  that  MEIMS  is 
beneficial  to  fleet  safety.  However,  as  with  any  new  major  software  program,  there  are 
areas  that  need  improvements. 

According  to  the  quantitative  survey  items,  the  MEIMS  prototype  type  received 
its  highest  ratings  in  ease  of  navigations  and  logical  format.  For  MEIMS  to  reach  its 
full  potential  the  following  items  must  be  completed: 

•  Incorporate  definitions  of  the  HFACS— ME  taxonomy  terms  into  the 
application,  this  will  enhance  the  users  understanding  of  the  taxonomy. 

•  Create  a  users  manual  and  help  function.  These  will  increase  the  user’s 
ability  to  understand  the  functions  of  MEIMS  and  increase  their  ability  to 
access  information  quickly. 

Increasing  the  users  knowledge  of  the  MEIMS  program,  and  all  it  functions,  as 
well  as  providing  more  detailed  information  on  the  HFACS— ME  taxonomy  will  make 
MEIMS  a  valuable  program  tool  for  any  fleet  unit. 

C.  RECOMMENDATIONS 

1.  Recommended  Prototype  MEIMS  Tool  Improvements 

•  Incorporate  HFACS— ME  definitions  within  MEIMS.  Better  descriptions 
of  the  HFACS— ME  causal  factors  will  also  improve  usability  and 
understanding. 

•  Include  a  detailed  description  for  each  mishap  to  augment  the  brief 
description. 

•  Include  mishap  data  prior  to  1989  and  from  2000  to  the  current 

•  Include  maintenance  related  Hazard  Reports. 
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•  Change  the  aircraft  identifier  to  include  aircraft  nickname  in  addition  to 
type/model  to  avoid  similar  names. 

•  Separate  AH-1  and  UH-1  into  two  categories,  due  to  the  aircraft’s 
differences. 

•  Provide  a  link  to  the  date-time-group  of  the  Mishap  Investigation  Report. 
This  will  give  the  user  the  option  to  obtain  the  complete  details 
surrounding  a  mishap. 

•  Incorporate  a  help  menu  within  the  program  and  a  user  manual.  This  will 
improve  the  end-users  knowledge  of  HF ACS— ME  and  make  MEIMS  a 
more  productive  tool. 

•  Expand  this  program  to  other  areas  with  in  the  armed  forces  where 
maintenance  is  performed. 

•  Develop  a  program  similar  to  MEIMS  using  the  aircrew  factors  taxonomy, 
HFACS. 

•  Make  MEIMS  available  over  the  World  Wide  Web.  This  will  ensure  that 
everyone,  regardless  of  location,  will  have  access  to  the  data. 

2.  The  Future  of  MEIMS 

MEIMS  has  developed  into  a  comprehensive  information  management  database 
tool  for  accessing  mishap  data  information  for  use  in  training,  analysis,  and 
investigations.  This  tool  has  the  potential  to  save  the  Naval  Aviation  money,  increase 
combat  and  personnel  readiness,  and,  ultimately,  save  lives.  Even  with  this  enormous 
potential,  this  program  is  only  in  its  infancy.  Expanding  the  MEIMS  concept  to  other 
areas  within  the  Service  where  maintenance  is  performed  could  further  enhance  the  safety 
impact.  Moreover,  MEIMS  could  be  further  expanded  to  civil  aviation  where  there  is 
also  a  record  of  human  error,  including  maintenance  related  human  error,  in  mishaps. 
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APPENDIX  A 


PROTOTYPE  MAINTENANCE  ERROR  INFORMATION  MANAGEMENT 
SYSTEM  (MEIMS)  TOOL  REVIEW 

1.  MAIN  MENU 

The  Main  Menu  appears  after  HFACS-(ME)  ICON  is  seleeted  (see  Figure  Al). 


'igure  Al.  Prototype  MEIMS  Tool  Main  Menu 


Select  the  “Query  Menu”  command  button  to  view  Query  Menu  (see  Figure  A2). 


2.  QUERY  MENU 


El  Queiy  Menu 


El 


Database  Query  Menu 


Multiple  Ccitena 


Figure  A2.  Query  Menu 


Select  the  “Multiple  Criteria”  command  button. 

The  Multiple  Criteria  Query  menu  appears  (see  Figure  A3). 
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Select  one  or  multiple  categories  in  the  drop  down  boxes  then  click  “View”,  and  the 
Summary  of  Mishap  form  appears  (see  Figure  A4). 

Selecting  the  Back  button  will  take  you  back  to  the  Main  Menu. 


Figure 
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■Mishap  #  12  -  US  Na\T  -  F14 
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Summary  of  Mishap  Form  with  F14'5€lected^aS^ircraft  Type  and  US 
Navy  as  Oj:gatuzation.^ 


Select  the  inner  right  arrow  aft^^ecord:”  to  yje^  additional  Mishap  Factors. 
Select  the  outer  right  arro^yMter  “Record:^^4dview  additional  Mishaps. 

Select  “Print”  to  view  a  printable  recoj 
Selecting  “Done”  will  take  you  back  to  the  Multiple  Criteria  Query  (see  Figure  A3). 
Selecting  the  “HFACS— ME  Summary”  from  the  Query  Menu  (see  Figure  A2)  will 
display  the  HFACS— ME  Summary  page  (see  Figure  A5). 
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figure  A5.  HFACS--ME  Summary-Eofm 


Selecting  one  or  multiple  criteria  from  th^jdrcJp  down  boxes  and  then  selecting  “Update’ 
will  recalculate  the  values  in  eachujof'tfie  factors  boxes. 

Clicking  on  any  factor  bpX'WtIl  bring  up  the  Summary  of  Mishap  Form  (see  Figure  A4) 
for  desired  use^ 

Selecting  Close  will  return  the  user  to  the  Query  Menu  (See  Figure  A2) 


3.  GRAPH  MENU 

Selecting  the  “Graph  Menu”  from  the  Main  Menu  (see  Figure  Al)  will  display  the  Graph 
Menu  (see  Figure  A6) 
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'Figure  A6.  Graph  Menu 

Select  one  of  the  Vadial  buttons  to  determine  the  value  of  the  X-axis,  and  one  of  the  radial 
buttons  to  determme  the  va^e  of  the  Y-axis. 

Select  “Show  Graph”  to  display  the  Graph  (see  Figure  A7). 

Checking  the  “Uselcodes  to  represent  X-axis/Y -axis  values  will  display  the  code  vice  the 
full  definition  in  the  X  and  Y  values  area  of  the  graph 

\ _ . 


Select  “Typ^X^hart”  or  “Dimemons”  to  change  the  type  and  stw  of  chart. 

Choosing  different  X  and  Y  values  and  then  selecting  “Updat^  button  will  update  the 
graph. 
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4.  REPORT  MENU 


Selecting  the  “Report  Menu”  from  the  Main  Menu  (see  Figure  Al)  will  display  the 
Report  Menu  (see  Figure  A8). 


S  Repofts  Menu 


Reports  Menu 


Cross-Tabbed  Reports 


Selecting  Mishap  Chronological  Listing  will  display  a  chronological  list  of  all  mishaps  in 
the  database. 
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Figure  A9.  Sample  Aircraft  Report 
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5.  INITIATE  INVESTIGATION  MENU 


Selecting  the  “Initiate  Investigation  from  the  Main  Menu  (see  Figure  Al)  will  open  up  a 
separate  Access  database  program  and  will  display  the  Mishap  Investigations  form  (see 
Figure  A 10) 


Figure  AlO.  Mishap  Investigations  Form 

Select  “Add”  to  start  the  investigation  decision  program  (see  Figure  Al  1). 


to  add  the  information.  When  done  select  “Next”  to  continue  entering  information  (see 
Figure  A 12). 
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Figure  A13.  Mishap  Category  Form 

Select  one  radial  button  for  the  appropriate  mishap  category  and  then  select  ‘'Next' 


Next”. 
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FirsI  Level  Faclois 


Was  there  a  Management  Condition  that  contributed  to  tNs  mishap? 

Examples: 

■  An  engine  change  is  performed  despite  a  high  sea  state 

•  A  manual  omits  a  step  calling  for  an  o-ring  to  be  installed. 

•  A  commander  does  not  ensure  that  personnel  wear  required  protective  gear 

•  A  technical  publication  does  not  specify  torque  requirements 

-  A  poor  component  layout  prohibits  direct  viewing  during  inspection. 

Click  yes  to  enter  a  factor  No  to  go  to  the  next  category. 


figure  A15.  First  Level  Management  Factor^ft^rm 

Select  “Yes”  to  add  a  factor  for  first  level  factor  that  is  stated  on  the 
A14). 


ee  Figure 


Select  “No”  if  there  is  no  factor  for  the  first  level  factor  that  is  stated  on  the  form  and 
return  to  the  Factor  Input  Menu  (see  Figure  A14). 
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Select 


Organ jyationai  Management  Conditions  -  Third  Level  Factors 


choose  a  third  level  factor  aod  provide  a  brief  narrative. 
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re  A17.  Third  Level  Manage^  Factor  form 


Select  the  appropriate  third  level  factor  from  ^ic  drop  down  box. 

Write  a  brief  description  of  the  factor. 

Select  “Next”  when  completed  filling  in  the  form.  The  program  will  take  the  user  back  to 
the  Factor  Input  Menu  (see  Figure  A14)  to  continue  entering  factors  until  all  factors  for 
the  mishap  are  entered  into  the  program.  Once  all  factors  are  entered  the  user  will  get  the 
finished  form  (see  Figure  A 18). 
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igure  A18.  Finished  For 

Select  “Back”  if  there  are  more  factors  to  add. 

Select  “Finished”  to  view  the  mishap  data  entered  (see  Figure  A 19). 
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Figure  A19.  Mishap  Data  Entered  Form 


Select  “Print”  to  view  the  Investigation  Mishap  Report  (see  Figure  A20). 

Selecting  “Save”  will  save  the  data  and  return  the  user  to  the  Mishap  Investigations  Form 
(see  Figure  A 10) 
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Figure  A20.  Mishap  Investigation  Report 
6.  ADMINISTRATION 

Selecting  “Administration”  on  the  Main  Menu  (see  Figure  Al)  will  bring  up  the  login  in 
form  (see  Figure  A21). 
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Figure  A21.  Adjnirtistrator  Login  Form 

Fill  in  password  and  press  “Ok”^ 


Figure  A22.  Successful  Login  Form 

Press  “OK”  to  view  the  Administration  Manage  Mishaps  Form  (see  Figure  A23) 
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Figure  A23.  Administration  Manage  Mishaps  F  rm 


Select  “Add”  10  add  a  mishap  and  its  associated  factors. 

Select  “KilF’Ao  delete  the  highlighted  record  and  its  associated  factors. 

Select  “View”  to  see  the  mishap  and  its  associated  factors  in  the  Mishap  Data  Form  (see 
Figure  A24). 
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Mishap  #377  -  US  Naw  -  H46 


Mishap  Date:  |l3-0ct-e9 


Location:  ^rrpariced  ' 


Organization:  [uSNary 


~31  Mishap  Class:  |<  $10000 


Aircraft  Model: 


«|  Mishap  Type:  [Aviation  Qroird  Mghap  *1 

Short  Deacrfition:  jaw's  right  rng  friger  was  crvahttVsevered  vvhtte  troii)teshoc>tii9  coriSol  system  j 


[Paciors  j  Detailed  Mishap  Descnpton  | 

Msho  Facte  Summanti 


Delete  Factor  Add  Factor 


Factor  Ooto  Detaiied  Factor  Descnpeon  I 


MAff /T  TECH  FaiiBd  to  Fotow  the  EST  GRD  Ti^  Procedixes 


3rd  Level:  j  InIraOon  ^  2nd  Level:  j 

Record;  l<  I  ll  T  •  (mI»  lol  of  I 


•  I  1st  Level:  [ 


Mantaner  Acts 


ishaps  Form 


Select  Print”  to  viewJheJnveStl^tion  Mishap  Report  (see  Figure  A18). 

Select  “Save^^lo'cEange  any  data  that  has  been  changed. 

Select  “Cancel”  to  either  close  the  form  or  to  cancel  any  changes  you  have  made. 


7.  EXIT 

Selecting  “Exit”  from  the  Main  Menu  (see  Figure  Al)  will  close  the  program. 
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APPENDIX  B 


PROTOTYPE  MAINTENANCE  ERROR  INFORMATION  MANAGEMENT 
SYSTEM  (MEIMS)  TOOL  EVALUATION 

Background.  Thank  you  for  participating  in  a  usability  study  (evaluation)  of  a 
prototype  tool  for  the  Maintenance  Error  Information  Management  System  (MEIMS). 
This  tool  was  developed  at  NPS  and  has  been  modified  based  upon  previous  usability 
studies.  This  study  is  being  conducted  by  Capt  Doug  Nelson,  USMC  as  part  of  a  thesis 
projeet  for  his  Master  of  Scienee  program  in  Information  Technology  Management. 
MEIMS  was  developed  to  address  and  identify  patterns  of  human  error  in  Naval  Aviation 
maintenance-related  aireraft  mishaps.  The  Human  Factors  Analysis  and  Classification 
System  Maintenance  Extension  (HFACS— ME)  taxonomy  is  the  foundation  of  MEIMS 
and  is  an  effective  method  for  elassifying  and  analyzing  the  presence  of  human  error  in 
maintenance  operations  leading  to  major  mishaps,  aceidents  of  lesser  severity,  incidents 
and  maintenanee  related  personal  injury  eases.  Given  the  eapability  of  the  previous 
MEIMS  systems  an  improved  information  management  system  was  needed  to  fiilly 
capture  Maintenanee  related  factors  and  bring  the  system  to  the  next  level. 

MEIMS  eaptures  maintenanee  error  data,  facilitates  the  identification  of  common 
maintenance  errors  and  associated  trends,  and  supports  understanding  of  how  to  identify 
human  errors  in  the  future.  The  target  audience  for  this  information  management  system 
tool  includes  safety  personnel  (data  entry  &  retrieval  by  unit  safety  offieers,  other  safety 
&  training  personnel,  maintenance  offieers,  maintenance  supervisors),  mishap 
investigators-for  data  retrieval  (Aireraft  Mishap  Board  members,  squadron  safety 
officers),  and  analysts  (from  the  Naval  Safety  Center,  the  eommand’s  safety  offieer  or 
one  from  its  higher  headquarters).  This  tool  can  directly  lead  to  a  deereased  mishap  rate 
and  overall  increased  mission  readiness  due  to  the  training  and  analysis  opportunity  it 
provides. 

Usability  Study.  You  will  be  given  a  paeket  of  instruetions  to  guide  you  through 
MEIMS.  You  will  be  asked  to  make  comments  on  the  effectiveness  and  usability  of  the 
prototype  system  during  your  testing  phase.  Additionally,  you  will  be  asked  to  complete 
an  “exit  survey”  after  eompletion  of  your  testing.  Questions  will  inelude  demographie 
information,  objective  questions  about  MEIMS  usability,  and  subjective  questions  and 
comments  for  areas  not  eovered  in  the  objeetive  seetion.  The  study  should  take  no  more 
than  20-30  minutes. 

Completion  of  Study.  Upon  completion  of  your  testing  and  survey  you  will  be 
asked  to  return  your  packet  of  instructions  to  Capt  Doug  Nelson.  Thank  you  again  for 
your  partieipation. 


Doug  Nelson 
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Instructions  for  Prototype  Maintenance  Error  Information  Management  System 

(MEIMS)  Tool  Evaluation 


Start-up 

1 .  Go  to  a  room  E-322. 

2.  When  Log-in  menu  appears,  Log-in  using  ASO  ID  and  password. 

3.  When  Desktop  (main  Icon  screen)  appears,  double  click  on  the  HFACS-(ME) 
icon.  This  will  start  the  MEIMS  application  using  Acess  2000  as  an  interface 
and  SQL  server  as  the  database  engine. 

Question  1:  Did  you  have  any  problems  accessing  the  program?  Y  /  N  (circle  one) 

If  so,  please  describe: 


Main  Menu 

4.  You  will  now  have  the  Main  Menu  displayed  with  the  Supersonic  Hornet  and 
Osprey  photos. 

5.  Note  the  six  categories  on  the  right  portion  of  the  screen. 

Question  2:  Is  the  terminology  clear  enough  to  understand  what  each  of  the  six 
command  buttons  does?  If  not,  what  could  be  changed  to  make  it  clearer? 


6.  Select  (click)  <Query  Menu> 

Query  Menu 

7.  Note  there  are  two  sections  on  the  Query  Menu.  Multiple  Criteria  and 
HE  ACS— ME  Summary 

8.  Select  (click)  Multiple  Criteria 

9.  Another  form  appears:  “Select  Multiple  Criteria  Query”.  Select  your  type 
aircraft,  and  then  select  <View  >.  The  “Mishap  Data”  form  appears.  Note  the 
number  of  maintenance  related  mishaps  is  on  the  bottom  of  the  outer  box  and  the 
number  of  factors  for  that  mishap  are  on  the  inner  box.  You  can  cycle  through 
mishaps  or  factors  by  selecting  the  (>)  to  the  right  of  the  number  box.  Review  the 
“Short  Description”  of  the  mishap  and  the  “Mishap  Factor  Summary”  .  Note 
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selecting  print  will  show  you  a  printable  report. 

Question  3:  What  aircraft  did  you  select? _ 

Question  4:  How  many  separate  mishaps  of  that  aircraft  type  are  in  the 
database? _ 

Question  5:  How  many  factors  are  there  for  the  first  mishap? _ 

10.  Select  “Done”  when  you  are  through. 

11.  You  are  now  free  to  choose  other  mishap  information  if  you  wish,  or  continue 
to  step  12. 

12.  Select  “<  Back”. 

13.  Select  “HFACS-ME  Summary” 

14.  HFACS  summary  by  factor  form  appears.  This  page  will  first  display  all  the 
mishaps  in  the  database  and  can  be  narrowed  down  by  the  user  with  the  drop 
down  boxes  on  the  right  and  then  selecting  “Update”  in  the  lower  right  hand 
comer.  The  form  displays  the  total  number  of  mishaps  (right  center),  as  well 
as  the  number  of  mishaps  for  each  factor  and  what  percent  that  factor  is  to  the 
total  number  of  mishaps. 

Question  6:  How  many  total  mishaps  are  in  the  database?  _ 

Question  7:  How  many  mishaps  have  a  level  one  category  of  Management  Conditions? 

Question  8:  What  is  the  percentage  of  mishaps  that  have  a  level  factor  of  Inadequate 

Design?  _ 

15.  Conduct  further  queries  as  desired  or  continue  to  step  16.  Note:  you  can 
click  on  any  factor  box  to  bring  up  detailed  information. 

16.  Select  “Close” 

17.  Select  “Return  to  Main  Menu” 

Graph  Menu 

18.  Select  “Graph  Menu” 

19.  The  Database  Graph  Menu  will  appear.  Select  “Mishap  Type”  for  the  X-axis 
value  and  “Organization”  for  the  Y-axis  value. 

20.  Select  “Show  Graph” 
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21.  A  graph  form  will  appear  for  the  values  seleeted.  You  ean  eliek  on  the  top 
eenter  of  any  bar  and  the  total  number  of  mishaps  for  your  seleeted  values 
will  show  up  on  the  bottom  left. 

Question  9:  How  many  total  Aviation  Ground  Mishaps  does  the  US  Navy  have? _ 

22.  You  are  free  to  play  with  the  graph  menu  or  go  onto  step  23. 

23.  Seleet  “Close” 

24.  Seleet  “Return  to  Main  Menu” 

Report  Menu 

25.  Seleet  “Report  Menu” 

26.  The  Report  Menu  Form  will  appear. 

27.  Seleet  “Mishaps  By  Class” 

28.  A  HFACS— ME  Summary  report  will  appear.  Mishaps  are  organized  by 
mishap  class  and  the  total  number  of  mishaps  will  be  displayed  for  that  class. 
Each  level  factor  will  display  the  total  number  of  mishaps  that  have  that 
factor  as  a  cause  of  the  mishap  as  well  as  a  percent  compared  to  total 
mishaps.  To  cycle  through  the  other  mishap  classes  click  the  right  arrow  (>) 
to  the  right  of  Page:  in  the  bottom  left  comer. 

Question  10:  How  many  total  class  B  mishaps  are  there?  _ 

29.  Select  “Close”  at  the  top  of  the  report  to  return  to  Report  Menu. 

30.  You  can  choose  other  reports  if  you  wish,  or  go  to  step  3 1 . 

3 1 .  Select  “Return  to  Main  Menu”. 

Initiate  Investigation 

The  following  scenario  will  help  you  in  completing  the  following  section; 
During  a  turnaround  inspection  the  plane  captain  fell  off  an  aircraft  and  broke 
his  arm  as  well  as  damaged  the  aircraft.  Due  to  his  injuries  the  plane  captain 
missed  two  workdays,  also  there  was  aircraft  damage  in  excess  of  $10,000. 
After  a  brief  investigation  of  this  incident  you  discover  that  the  maintenance 
chief  directed  the  plane  captain  to  get  the  turnaround  done  despite  the  pouring 
rain  outside.  You  also  discover  that  the  plane  captain  had  duty  the  night 
before  and  had  been  up  all  night. 
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32.  Select  “Initiate  Investigation” 

33.  A  new  database  program  will  open  in  Access  2000  and  the  Mishap 
Investigation  form  will  appear.  This  section  is  for  the  user  to  enter  mishap 
data  using  the  HFAC-ME  taxonomy.  Select  “Add” 

34.  Add  new  mishap  wizard  will  appear.  Fill  in  the  appropriate  information 
using  your  personal  information  and  the  scenario  above.  You  can  fill  in  the 
same  information  for  short  and  long  description.  Click  “Next  >”  when  done. 

35.  Mishap  Class  screen  appears;  check  the  appropriate  injuries  and  cost  boxes. 
Click  “Next  >”  when  done. 

36.  Mishap  Categories  screen  appears;  check  the  appropriate  category.  Click 
“Next  >”  when  done 

37.  Factor  Input  screen  appears.  You  are  now  going  to  be  guided  through  the 
HFACS— ME  taxonomy  with  a  series  of  questions  until  all  possible  factors  for 
your  mishap  have  been  entered.  Click  “Next  >”  to  continue. 

38.  At  the  3’^'^  level  screen  you  will  select  one  of  the  third  level  factors  and  write  a 
brief  description  of  the  factor  that  contributed  to  the  mishap.  Click  “Next  >” 
when  done. 

39.  Factor  Input  screen  appears.  Click  “Next  >”,  you  will  again  be  asked  if  there 
were  any  Management  Conditions.  If  you  have  no  more  Management 
Conditions  to  enter  select  “NO”.  Click  “Next>”  and  continue  this  cycle  until 
all  factors  have  been  entered  for  your  mishap.  Notice  that  a  check  will  appear 
before  each  first  level  factor  when  you  have  completed  that  factor. 

40.  When  all  factors  are  entered  Click  “Next>”.  You  will  be  at  the  finished 
screen  and  select  “Finished” 

41.  The  Mishap  Data  form  will  now  have  the  data  you  entered.  Click  on  “Prinf’ 
in  the  lower  left  section  of  the  screen. 

42.  The  data  is  now  arranged  in  report  format  suitable  for  printing. 

Question  11.  Do  you  feel  this  section  helped  you  in  identifying  all  possible  factors  for 
this  mishap?  Yes/No  (circle  one),  If  not  please  describe. 
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43.  Select  “Close”  on  the  top  left. 

44.  Select  “Cancel” 

45.  Select  “Done’ 

Administration 

50.  The  Administration  section  is  for  Adding,  Deleting  or  modifying  existing 
data  in  the  database  and  is  not  a  testable  portion. 

Exit 

5 1 .  Select  “Exit” 

52.  Select  “Yes” 

53.  Please  fill  out  an  Exit  Survey  Questionnaire. 
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APPENDIX  C 


PROTOTYPE  MAINTENANCE  ERROR  INFORMATION  MANAGEMENT 
SYSTEM  (MEIMS)  TOOL  EXIT  SURVEY 

User’s  Impression  of  the  Maintenanee  Error  Information  Management  System  (MEIMS) 

Prototype  Tool 

Purpose:  This  survey  evaluates  a  user’s  overall  satisfaction  of  the  Maintenance  Error 
Information  Management  System  (MEIMS)  prototype  tool.  It  consists  of  three  parts. 

Part  I:  Demographic  Information.  Part  I  provides  the  user’s  aviation 
background,  computer  experience,  and  availability  of  software  and  hardware  systems 
used  in  the  Navy  and  Marine  Corps. 

Part  II:  User  Satisfaction  with  the  Four  Sections  of  the  MEIMS  Prototype  Tool. 
Part  II  deals  directly  with  user  feedback  as  they  use  the  prototype  tool. 

Part  III:  User  Overall  Satisfaction  with  the  MEIMS  Prototype  Tool.  Part  III 
allows  users  to  give  general  feedback  about  the  prototype  tool. 


Part  I.  Demographic  Information 

Follow  the  instructions  after  each  numbered  question  or  statement. 

1.  I  am  currently/was  attached  to  a  command  that  primarily  performs  maintenance 
(military  and/or  civilian)  at  the: 

(Select  one  from  the  list  and  check  the  box) 

□  Squadron  Level 

□  Intermediate  Level  (AIMD) 

□  Depot  Level  (NADEP) 

□  Command  does  not  perform  aircraft  maintenance 

□  Other  (describe  if  other) _ 

2.  How  long  have  you  been  using  a  computer? 

(Select  one  from  the  list  and  check  the  box) 

□  Less  than  one  month 

□  One  month  to  less  than  one  year 

□  One  year  to  less  than  two  years 

□  Two  years  or  more 
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3.  What  software  do  you  normally  use? 

(Check  all  boxes  that  apply) 

□  Microsoft  Office  (Word,  PowerPoint,  Excel,  Access) 
What  version? 

(Check  all  boxes  that  apply) 

□  97 

□  2000 

□  not  sure  of  version 

□  other  (describe  if  other) _ 


4.  What  software  application  categories  are  you  familiar  with? 
(Check  all  boxes  that  apply) 

□  Word  Processing  (MS  Word,  Word  Perfect,  Word  Pro...) 

□  Spreadsheet  (Excel,  Lotus  123,  Quattro  Pro...) 

□  Presentations  (PowerPoint,  Harvard  Graphics...) 

□  Graphic  Software  (Corel  Draw,  Adobe  Photoshop...) 

□  E-Mail  (Outlook,  Eudora,  AOL...) 

□  Database  (Access,  DBase...) 


5.  What  computer  operating  systems  do  you  use? 
(Check  all  boxes  that  apply) 

□  Windows  (3.1,  95,  98,2000) 

□  Windows  NT 

□  Macintosh 

□  UNIX 

□  Linux 

□  Other  (describe  if  other) _ 


Part  II.  User  Satisfaction  with  the  Four  Sections  of  the  MEIMS  Prototype  Tool 

Select  the  category  that  best  matches  your  impression  of  each  of  the  below  categories 
(and  check  the  box). 


Strongly 

Agree 

Agree 

Neutral 

Disagree 

Strongly 

Disagree 

I  feel  the  information 
on  the  MEIMS  tool  was 

in  a  logical  form 

(comments) 

□ 

□ 

□ 

□ 

□ 
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Strongly  Agree  Neutral  Disagree  Strongly 
Agree  Disagree 


I  found  the  MEIMS 
tool  easy  to  navigate 

(comments) 

□ 

□ 

□ 

□ 

□ 

My  tour  of  the  MEIMS 
tool  was  very  interesting 

(comments) 

□ 

□ 

□ 

□ 

□ 

The  information  presented  on 
the  MEIMS  tool  is  relevant  to 
maintenance  operations 

(comments) 

□ 

□ 

□ 

□ 

□ 

The  concept  of  the  MEIMS 
tool  is  a  good  one. 

(comments) 

□ 

□ 

□ 

□ 

□ 

I  found  the  investigation 
tool  to  be  helpful 

(comments) 

□ 

□ 

□ 

□ 

□ 
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Part  III.  User  Overall  Satisfaction  with  the  MEIMS  Prototype  Tool 

Please  make  any  comments  you  may  have  on  the  MEIMS  Prototype  Tool  not  reflected  in 
your  comments  in  sections  1  and  2.  Please  use  back  of  paper  if  you  need  more  room. 

The  most  positive  aspects  of  the  MEIMS  prototype  tool  were: 


The  most  negative  aspects  of  the  MEIMS  prototype  tool  were: 


I  would  make  these  changes  (if  any)  to  the  MEIMS  prototype  tool: 


I  would  make  the  following  changes  (if  any)  to  the  investigation  portion: 


Any  additional  comments: 


Thank  you!  Your  participation  is  greatly  appreciated! 
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APPENDIX  D 


PROTOTYPE  MAINTENANCE  ERROR  INFORMATION 
MANAGEMENT  SYSTEM  (MEIMS)  ENTITY  RELATIONSHIP  DIAGRAM 


Figure  DI.  Entity  Relationship  Diagram 
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APPENDIX  E 


PROTOTYPE  MAINTENANCE  ERROR  INFORMATION 
MANAGEMENT  SYSTEM  (MEIMS)  METADATA 

Table  6:  HFACS-ME  Tables 


Table  Name 

Number  of  Columns 

Primary  Key 

tblMishaps 

11 

MishapID 

tblMishapF actors 

5 

FactorlD 

tblF actors 

6 

3rdLevelCode 

tblAircraft 

3 

AircraftT  ypeModel 

tblMishapType 

2 

MishapTypeCode 

tblMishapClass 

2 

MishapClassCode 

tblOrganization 

3 

OrgID 

tblMishapLocation 

3 

MishapLocationID 

tblDatabaseType 

1 

DatabaseType 

Table  7:  HFACS-ME  Columns 


Column  Name 

Table  Name 

Data  Type 

Length 

MishapID 

TblMishaps 

int 

4 

MishapDate 

TblMishaps 

datetime 

8 

Aircraft  FK 

TblMishaps 

nvarchar 

50 

Class  FK 

TblMishaps 

nvarchar 

5 

TypeFK 

TblMishaps 

nvarchar 

5 

LocationID  FK 

TblMishaps 

nvarchar 

50 

OrgID  FK 

TblMishaps 

nvarchar 

50 

ShortDescription 

TblMishaps 

nvarchar 

255 

LongDescription 

TblMishaps 

nvarchar 

4000 

Underinvestigation 

TblMishaps 

bit 

1 

DatabaseType 

TblMishaps 

nvarchar 

5 

FactorlD 

TblMishapFactors 

int 

4 

MishapIDFK 

TblMishapFactors 

int 

4 

FactorSummary 

TblMishapFactors 

nvarchar 

255 

3rdLevelCode_FK 

TblMishapF  actors 

nvarchar 

3 

F  actorDescription 

TblMishapFactors 

nvarchar 

4000 

3rdLevelCode 

TblFactors 

nvarchar 

5 

3rdLevelDesc 

TblFactors 

nvarchar 

50 

2ndLevelCode 

TblFactors 

nvarchar 

5 

2ndLevelDesc 

TblFactors 

nvarchar 

50 

1  stLevelCode 

TblFactors 

nvarchar 

5 

IstLevelDesc 

TblFactors 

nvarchar 

50 
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AircraftT  ypeModel 

TblAireraft 

nvarehar 

15 

AircraftCategory 

TblAireraft 

nvarehar 

50 

AircraftDescription 

TblAireraft 

nvarehar 

255 

MishapTypeCode 

TblMishapType 

nvarehar 

5 

MishapTypeDefinition 

TblMishapType 

nvarehar 

255 

MishapClassCode 

TblMishapClass 

nvarehar 

2 

MishapClassDefmition 

TblMishapClass 

nvarehar 

255 

OrgID 

tblOrganization 

nvarehar 

10 

OrgName 

tblOrganization 

nvarehar 

50 

DatabaseType 

tblOrganization 

nvarehar 

1 

MishapLocationID 

tblMishapLoeation 

nvarehar 

50 

MishapLocation 

tblMishapLoeation 

nvarehar 

50 

DatabaseType 

tblMishapLoeation 

nvarehar 

1 

DatabaseType 

tblDatabaseType 

nvarehar 

50 
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